Object detection and image cropping using a multi-detector approach

ABSTRACT

According to an exemplary embodiment, a method for pre-cropping digital image data includes: dividing the digital image into segments; computing a color value distance between corresponding pixels of neighboring segments of the digital image; comparing the color value distance(s) against a minimum color distance threshold; clustering neighboring segments having a color value distance less than or equal to the minimum color distance threshold; computing a connected structure based on the clustered segments; computing a polygon bounding the connected structure; comparing a fraction of segments included in the connected structure and the polygon, relative to a total number of segments in the digital image, to a minimum included segment threshold; and in response to determining the fraction of segments in the connected structure and the polygon, relative to the total number of segments meets or exceeds a minimum included segment threshold, cropping the digital image based on edges of the polygon.

FIELD OF INVENTION

The present invention relates to image capture and image processing. Inparticular, the present invention relates to capturing and processingdigital images using a mobile device, with special emphasis on detectingobjects such as documents within the image and cropping the image so asto remove background and/or other objects therefrom.

RELATED APPLICATIONS

This application is related to U.S. Pat. No. 9,779,296, granted Oct. 3,2017 and entitled “Content-Based Detection And Three DimensionalGeometric Reconstruction Of Objects In Image And Video Data.”; U.S. Pat.No. 9,760,788, granted Sep. 12, 2017 and entitled “Mobile DocumentDetection And Orientation Based On Reference Object Characteristics”;U.S. Pat. No. 9,355,312, granted May 31, 2016 and entitled “Systems AndMethods For Classifying Objects In Digital Images Captured Using MobileDevices”; U.S. Pat. No. 9,208,536, granted Dec. 8, 2015 and entitled“Systems And Methods For Three Dimensional Geometric Reconstruction OfCaptured Image Data;” and U.S. Pat. No. 8,855,375, granted Oct. 7, 2014and entitled “Systems and Methods for Mobile Image Capture andProcessing”; each of which is herein incorporated by reference in itsentirety.

BACKGROUND OF THE INVENTION

As imaging technology improves and promulgates, an increasing number ofapplications and contexts are being explored to expand and improve uponaccomplishments achieved to-date. In the particular context of objectdetection, and the related determination and/or extraction ofinformation relating to recognized objects, significant work performedto-date affords a plethora of conventional approaches generally capableof performing desired detection and extraction. However, theseconventional approaches generally rely on highly tuned image quality andenvironmental control in order to function properly.

Of course, in all circumstances it is ideal for capture conditions to beoptimal for the particular purpose to which the captured image will beapplied. Images should be free of blur and distortions, the image shoulddepict the entire object and ideally the object should comprise asignificant portion (e.g. at least 50%) of the total area of thecaptured image. Illumination should be well-balanced so as to clearlydepict various features (e.g. different colors, textures, connectedcomponents, boundaries, edges, etc.) present in the image foregroundand/or background, without over or undersaturating the image and losingassociated detail useful for distinguishing the object from thebackground and/or individual features of the object. The image should becaptured with sufficient resolution and color depth so as to allowfacile identification of individual features.

Conventional flat-bed scanners, multifunction printer (MFP) devices, andthe like advantageously yield images with high resolution, and typicallylack distortions associated with conventional cameras and similaroptical sensors (particularly perspective distortion arising fromorientation of the camera relative to the capture object, and warping or“fishbowl” type effects inherent to images captured with conventionalcameras). Images captured using conventional flat-bed scanners, MFPs,etc. also generally exhibit a simple background with knowncharacteristics (e.g. a pure white background) that facilitatesdistinguishing the background from the object (or foreground) depictedin the image.

However, images captured using flat-bed scanners, MFPs, etc. tend toexhibit oversaturation, which can make detecting the edges of objects(or equivalently, transition from the image background to the object orimage foreground) difficult or impossible. Similarly, smaller objectssuch as identification cards, credit cards, business cards, receipts,etc. may represent only a relatively small portion (e.g. about 20% orless) of the total image area, and/or may be located in an unexpectedposition and/or orientation within the image. The relatively small sizecauses difficulty in object detection, since there is relatively lessinformation available for use in determining characteristics of theforeground and distinguishing the same from the background.Unknown/unexpected orientation and position can similarly frustrate theprocess of determining background and foreground characteristics, forinstance because it can be difficult to sample characteristics from apredetermined region with confidence the characteristics trulycorrespond only to the background or foreground/object.

Images captured using cameras, especially cameras or similar opticalsensors included in mobile devices such as smartphones, tablets,personal digital assistants, drones, and the like exhibit a differentset of advantages and challenges with respect to object detection andimage cropping. For instance, cameras may advantageously capture aplurality of images of an object in rapid sequence (e.g. using a videocapture functionality) which increases the amount of informationavailable about both the object and the image background. However, asnoted above images captured using cameras and the like are characterizedby inherent distortion and/or warping. Moreover, in general the captureconditions surrounding use of a camera are more variable and lesscapable of control than corresponding conditions when capturing imagedata using a flat-bed scanner, MFP, or the like. Lighting conditions,camera motion and/or relative position to the object, and presence ofcomplex backgrounds are particularly challenging aspects of capturingimage data using a camera and which frustrate or even defeat the abilityto perform desired processing of the captured image data, e.g. objectdetection and accurate cropping.

Accordingly, it would be beneficial to provide new and improvedtechniques for detecting objects within image data and cropping suchimages, in a manner that can account for undesirable artifacts includingbut not limited to: glare, perspective distortion, image warping, lowcontrast between image foreground and background (e.g. due tooversaturation), relatively low total area of the image foreground,undesirable location/orientation of an object within the image, presenceof complex background and/or foreground, and any combination thereof. Itwould be even more beneficial to provide techniques capable ofaddressing challenges that arise in the context of scanned images aswell as artifacts that arise in the context of camera-captured images,so as to enable processing thereof in accordance with a singleoverarching protocol that can accurately and precisely identify objectsand crop images in a reproducible manner, regardless of the source ofthe input image data.

SUMMARY

According to one embodiment, a computer-implemented method forpre-cropping digital image data to remove background from the digitalimage data includes: dividing the digital image into a plurality ofsegments; computing a color value distance between corresponding pixelsof one or more pairs of neighboring segments of the digital image;comparing the color value distance computed for some or all of the pairsof neighboring segments against a minimum color distance threshold;clustering all pairs of neighboring segments having a color valuedistance less than or equal to the minimum color distance threshold;computing a connected structure based on a shape of the clusteredsegments; computing a polygon bounding the connected structure;comparing a fraction of segments included in both the connectedstructure and the polygon, relative to a total number of segments in thedigital image, to a minimum included segment threshold; and in responseto determining the fraction of segments included in both the connectedstructure and the polygon, relative to the total number of segments inthe digital image meets or exceeds a minimum included segment threshold,cropping the digital image based on edges of the polygon.

Other aspects and embodiments of the present invention will becomeapparent from the following detailed description, which, when taken inconjunction with the drawings, illustrate by way of example theprinciples of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a network architecture, in accordance with oneembodiment.

FIG. 2 shows a representative hardware environment that may beassociated with the servers and/or clients of FIG. 1 , in accordancewith one embodiment.

FIG. 3A is a photographic representation of a low-contrast imagecaptured using a conventional flat-bed scanner, according to oneembodiment.

FIG. 3B is a photographic representation of an image captured using aconventional flat-bed scanner, the image having a small object depictedtherein, and the object being located in an unconventional location andorientation, according to one embodiment.

FIG. 3C is a photographic representation of an image captured using acamera and exhibiting glare, according to one embodiment.

FIG. 3D is a photographic representation of an image captured using acamera and exhibiting distortion, according to one embodiment.

FIGS. 3E-3H are photographic representations of images having a complexbackground, according to various embodiments. In FIG. 3E, an object isdepicted on a complex background having three different regions eachwith a unique color profile, according to one embodiment. In FIG. 3F, anobject is depicted on a complex background including objects of asimilar type as the object desired for detection, according to oneembodiment. In FIG. 3G an object is depicted on a complex backgroundhaving six different regions each with a unique color profile, accordingto one embodiment. In FIG. 3H, an object is depicted with portionsthereof obscured by another object, and other portions thereof obscuredby shadow, according to one embodiment.

FIG. 4A is a photographic representation of an input image capturedusing a camera and depicting a document as the object for detection,according to one embodiment.

FIG. 4B depicts a graphical representation of a result of a line segmentdetector applied to the image shown in FIG. 4A, according to oneembodiment.

FIG. 5 is a flowchart of a method of detecting objects within digitalimage data, according to one embodiment.

FIG. 6 is a flowchart of a method of detecting objects within digitalvideo data in real-time or near-real time, according to one embodiment.

FIG. 7 is a flowchart of a method for detecting objects according tocolor information, according to one embodiment.

FIG. 8 is a flowchart of a method for detecting objects within digitalimage data based at least in part on color transitions within thedigital image data.

FIG. 9 is a flowchart of a method for detecting objects using aline-segmentation approach, according to one embodiment.

FIG. 10 is a flowchart of a method for pre-cropping digital image datadepicting an object so as to reduce the amount of background in thepre-cropped digital image, according to one embodiment.

DETAILED DESCRIPTION

The following description is made for the purpose of illustrating thegeneral principles of the present invention and is not meant to limitthe inventive concepts claimed herein. Further, particular featuresdescribed herein can be used in combination with other describedfeatures in each of the various possible combinations and permutations.

Unless otherwise specifically defined herein, all terms are to be giventheir broadest possible interpretation including meanings implied fromthe specification as well as meanings understood by those skilled in theart and/or as defined in dictionaries, treatises, etc.

It must also be noted that, as used in the specification and theappended claims, the singular forms “a,” “an” and “the” include pluralreferents unless otherwise specified.

As referred-to herein, it should be understood that the term “connectedcomponent” refers to any structure within an image, preferably a bitonalimage that is formed from a contiguous set of adjacent pixels. Forexample connected components may include lines (e.g. part of adocument's structure such as field boundaries in a form), graphicalelements (e.g. photographs, logos, illustrations, unique markings,etc.), text (e.g. characters, symbols, handwriting, etc.) or any otherfeature depicted in a bitonal image. Accordingly, in one embodiment aconnected component may be defined within a bitonal image according tothe location of the various pixels from which the component is formed.

The term “image feature” is to be understood as inclusive of connectedcomponents, but also includes such components as may be defined withincolor spaces other than a bitonal image. Thus, an image feature includesany structure of an image that is formed from a contiguous set ofadjacent pixels. The image feature may be defined according to thelocation of constituent pixels as noted above for connected components,but may also include other information such as intensity information(e.g. in one or more color channels). In various embodiments, “imagefeatures” may include any exemplary type or form as described in U.S.Pat. No. 9,779,296, granted Oct. 3, 2017 and entitled “Content-BasedDetection And Three Dimensional Geometric Reconstruction Of Objects InImage And Video Data.” In one particular embodiment, image features maycomprise “boilerplate content” as defined in U.S. Pat. No. 9,779,296. Instill more embodiments, image features may be or include “maximum stableregions” as known in in the art, e.g.https://en.wikipedia.org/wiki/Maximally stable extremal regions (lastvisited Nov. 20, 2017).

Image “foreground” should be understood as referring to an object ofinterest, e.g. an object subject to detection within the image data.Exemplary objects include documents, vehicles or parts thereof, persons,faces, or any other object capable of being described according todefining visual characteristics such as shape and color.

Conversely, image “background” as utilized herein refers to anynon-foreground object, texture, scene, and combinations thereof. Inother words, the background includes everything depicted in the imagedata other than the object of interest.

A “complex” background or foreground is to be understood as a portion ofan image that exhibits/includes either: (1) significant variation incolors, textures, and/or objects; (2) features likely to generate falsepositive edge/boundary identification, such as straight lines arrangedin a polygonal shape (such as a rectangle for a document depicted and/orforming part of an image background), curved lines arranged according toa particular object profile (including simple shapes such as a circle orsphere, or more complex arrangements such as a car, person, face, etc.);or (3) combinations thereof. Exemplary “complex backgrounds” include ascene, such as an environmental scene, social scene; a document such asa poster, magazine cover, photograph, form, sheet of lined or graphpaper; textures such as wood grain, marble, carpet, fabric, etc.;patterns such as lines, grids, spirals, mosaics, zig-zags, or anyequivalent thereof that would be appreciated by a person having ordinaryskill in the art upon reading the present descriptions. Severalexemplary embodiments of images having “complex backgrounds” are shownin FIGS. 3E-3H.

An “edge” or “border” of an object shall be understood as referring to atransition from image background to foreground, or equivalently fromimage background to the object of interest. Transitions from the objectof interest/foreground to the background are also “edges” or “borders.”

Image “distortion,” also referred to herein as “perspective distortion”is to be understood as artifacts associated with capturing an image ofan object at a capture angle deviating from normal, with respect to theobject. Distortion causes the image of the object to depict a differentshape than the true object's configuration. For example, an image of arectangle exhibiting substantial perspective distortion may appear as atrapezoid. Generally, perspective distortion appears as a lineartransformation in the captured image, although some forms of perspectivedistortion may appear as curved. In accordance with one embodiment, animage exhibiting distortion is shown in FIG. 3D.

Image “warping,” characterized by “fishbowl” effects and the like, asdiscussed herein, refer to radial distortion such as barrel distortion,pincushion distortion, mustache distortion, and the like which arisenaturally from the symmetry of a photographic lens, and particularlywith the use of zoom. Generally, warping effects appear as curvature andmagnification/demagnification of portions of the image.

“Oversaturated” and “low contrast” images are to be understood asreferring to images in which the amount of contrast between foregroundand background elements is sufficiently low that conventional imageprocessing techniques, such as those that attempt to directly detect theobject within the image, are incapable of reliably discriminating theboundaries between the foreground and background. In one exemplaryinstance, images with an average pixel brightness in a local regionabove a predetermined threshold value of e.g. 220 (on a scale of 0-255)should be considered “oversaturated,” while images having a contrastdifferent of about 50 or less, preferably in a range from about 15 to50, should be considered “low contrast.”

Oversaturation and/or low contrast most typically arises from use ofconventional flat-bed scanners, MFP devices, or the like, and/orinappropriate use of a flash setting with a conventional camera.Oversaturation generally appears as an excess of white/bright intensity,while low contrast may appear as any combination of similar colors,pixel intensities, etc. that result in sufficient similarity between theforeground and background so as to make distinguishing therebetweenexceedingly difficult. In accordance with one exemplary embodiment alow-contrast image is shown in FIG. 3A.

“Glare” may appear substantially as an oversaturated region of an image,and is generally caused by presence of a light source illuminating theregion (and any corresponding object that may be present therein) at aparticular angle such that much of the light is reflected/deflecteddirectly toward the camera lens. Conversely, “shadows” may appearsubstantially as an undersaturated region of an image, generally causedby a lack of sufficient illumination in the region, e.g. as occurs wherea foreign object is placed in the path between a light source and thetarget object being captured. Another problem is that the lightingdifferences between the shadow- and non-shadow regions tend to create astrong edge that is often mistaken for a document boundary usingconventional detection techniques.

The present application refers to image processing, and addresses theproblems associated with attempting to detect objects of interest withincomplex and/or difficult image data, and crop such images to exclude thecomplex/difficult background but retain the object of interest.

In one general embodiment, a computer-implemented method of detectingobjects within digital image data includes: receiving digital imagedata; analyzing the digital image data using one or more detectors, eachdetector being independently configured to detect objects within digitalimages according to a unique set of analysis parameters; determining aconfidence score for each of a plurality of analysis results produced bythe one or more detectors; selecting the analysis result having ahighest confidence score among the plurality of analysis results as anoptimum object location result; and one or more of: outputting, based onthe optimum object location result, a projected location of one or moreedges of the object to a memory; and displaying, based on the optimumobject location result, a projected location of the one or more edges ofthe object on a display.

In another general embodiment, a computer-implemented method ofdetecting objects within digital video data includes: defining ananalysis profile comprising an initial number of analysis cyclesdedicated to each of a plurality of detectors, each detector beingindependently configured to detect objects according to a unique set ofanalysis parameters; receiving a plurality of frames of digital videodata, the digital video data depicting an object; analyzing theplurality of frames using the plurality of detectors and in accordancewith the analysis profile, wherein analyzing the plurality of framesproduces an analysis result for each of the plurality of detectors;determining a confidence score for each of the analysis results; andupdating the analysis profile by adjusting the number of analysis cyclesdedicated to at least one of the plurality of detectors based on theconfidence scores.

In yet another general embodiment, a computer-implemented method ofdetecting objects within digital image data includes: receiving digitalimage data; analyzing the digital image data using one or more colortransition detectors, each detector being independently configured todetect objects within digital images according to a unique set ofanalysis parameters; determining a confidence score for each of aplurality of analysis results produced by the one or more colortransition detectors; selecting the analysis result having a highestconfidence score among the plurality of analysis results as an optimumobject location result; and one or more of: outputting, based on theoptimum object location result, a projected location of one or moreedges of the object to a memory; and rendering, based on the optimumobject location result, a projected location of the one or more edges ofthe object on a display.

In still yet another general embodiment, a computer-implemented methodof detecting objects within digital image data includes: receivingdigital image data depicting an object; analyzing the digital image datausing one or more line segment detectors, each detector beingindependently configured to detect objects within digital imagesaccording to a unique set of analysis parameters; determining aconfidence score for each of a plurality of analysis results produced bythe one or more line segment detectors; selecting the analysis resulthaving a highest confidence score among the plurality of analysisresults as an optimum object location result; and one or more of:outputting, based on the optimum object location result, a projectedlocation of one or more edges of the object to a memory; and rendering,based on the optimum object location result, a projected location of theone or more edges of the object on a display.

According to another general embodiment, a computer-implemented methodfor detecting objects within digital image data based at least in parton color transitions within the digital image data includes: receivingor capturing a digital image depicting an object; sampling colorinformation from a first plurality of pixels of the digital image,wherein each of the first plurality of pixels is located in a backgroundregion of the digital image; optionally sampling color information froma second plurality of pixels of the digital image, wherein each of thesecond plurality of pixels is located in a foreground region of thedigital image; generating or receiving a representative background colorprofile, the representative background color profile being based on thecolor information sampled from the first plurality of pixels; generatingor receiving a representative foreground color profile based on thecolor information sampled from the second plurality of pixels and/or thecolor information sampled from the first plurality of pixels; assigningeach pixel within the digital image a label of either foreground orbackground using an adaptive label learning process; binarizing thedigital image based on the labels assigned to each pixel; detecting oneor more contours within the binarized digital image; and defining one ormore edges of the object based on the detected contour(s).

According to yet another general embodiment, a computer program productfor detecting objects within digital image data based at least in parton color transitions within the digital image data includes a computerreadable storage medium having embodied therewith computer readableprogram instructions. The program instructions are configured to cause aprocessor, upon execution of the computer readable program instructions,to perform a method comprising: receiving or capturing a digital imagedepicting an object; sampling, using the processor, color informationfrom a first plurality of pixels of the digital image, wherein each ofthe first plurality of pixels is located in a background region of thedigital image; optionally sampling, using the processor, colorinformation from a second plurality of pixels of the digital image,wherein each of the second plurality of pixels is located in aforeground region of the digital image; generating, using the processor,or receiving, by the processor, a representative background colorprofile, the representative background color profile being based on thecolor information sampled from the first plurality of pixels;generating, using the processor, or receiving, by the processor, arepresentative foreground color profile based on the color informationsampled from the second plurality of pixels and/or the color informationsampled from the first plurality of pixels; assigning, using theprocessor, each pixel within the digital image a label of eitherforeground or background using an adaptive label learning process;binarizing, using the processor, the digital image based on the labelsassigned to each pixel; detecting, using the processor, one or morecontours within the binarized digital image; and defining, using theprocessor, one or more edges of the object based on the detectedcontour(s).

In yet further general embodiments, a system for detecting objectswithin digital image data based at least in part on color transitionswithin the digital image data includes: a processor, and logicintegrated with and/or executable by the processor to cause theprocessor to: receive or capture a digital image depicting an object;sample color information from a first plurality of pixels of the digitalimage, wherein each of the first plurality of pixels is located in abackground region of the digital image; optionally sample, using theprocessor, color information from a second plurality of pixels of thedigital image, wherein each of the second plurality of pixels is locatedin a foreground region of the digital image; generate or receiving arepresentative background color profile, the representative backgroundcolor profile being based on the color information sampled from thefirst plurality of pixels; generate or receive a representativeforeground color profile based on the color information sampled from thesecond plurality of pixels and/or the color information sampled from thefirst plurality of pixels; assign each pixel within the digital image alabel of either foreground or background using an adaptive labellearning process; binarize the digital image based on the labelsassigned to each pixel; detect one or more contours within the binarizeddigital image; and define one or more edges of the object based on thedetected contour(s).

In still yet further general embodiments, a computer-implemented methodof pre-cropping a digital image includes: downscaling a received digitalimage to a predetermined resolution; de-blurring the downscaled image toreduce color variations within the downscaled image; dividing thede-blurred image into a plurality of segments; computing color distancesbetween neighboring ones of the plurality of segments, wherein thedistances are color value distances between central pixels of theneighboring segments; comparing the color distances between each segmentand each corresponding neighboring segment against a predeterminedthreshold; clustering segments having color distances less than thepredetermined threshold to form a connected structure; computing apolygon bounding the connected structure; determining whether a fractionof the segments included within both the connected structure and thepolygon is greater than a predetermined threshold; and in response todetermining the fraction of the segments included within both theconnected structure and the polygon is greater than the predeterminedthreshold, cropping the digital image based on the edges of the polygon;and in response to determining the fraction of the segments includedwithin both the connected structure and the polygon is less than orequal to the predetermined threshold, repeating the computing colordistances, comparing color distances, clustering segments, computingconnected structure, and computing a polygon using a less restrictivecolor difference threshold.

Of course, the foregoing embodiments may also be implemented as systemsand/or computer program products, in various approaches consistent withthe inventive concepts presented herein.

General Mobile Networking and Computing Concepts

As understood herein, a mobile device is any device capable of receivingdata without having power supplied via a physical connection (e.g. wire,cord, cable, etc.) and capable of receiving data without a physical dataconnection (e.g. wire, cord, cable, etc.). Mobile devices within thescope of the present disclosures include exemplary devices such as amobile telephone, smartphone, tablet, personal digital assistant, iPod®,iPad®, BLACKBERRY® device, etc.

However, as it will become apparent from the descriptions of variousfunctionalities, the presently disclosed mobile image processingalgorithms can be applied, sometimes with certain modifications, toimages coming from flat-bed scanners and multifunction peripherals(MFPs). Similarly, images processed using the presently disclosedprocessing algorithms may be further processed using conventionalflat-bed scanner processing algorithms, in some approaches.

Of course, the various embodiments set forth herein may be implementedutilizing hardware, software, or any desired combination thereof. Forthat matter, any type of logic may be utilized which is capable ofimplementing the various functionality set forth herein.

As will be appreciated by one skilled in the art, aspects of the presentinvention may be embodied as a system, method or computer programproduct. Accordingly, aspects of the present invention may take the formof an entirely hardware embodiment, an entirely software embodiment(including firmware, resident software, micro-code, etc.) or anembodiment combining software and hardware aspects that may allgenerally be referred to herein as “logic,” “circuit,” “module” or“system.” Furthermore, aspects of the present invention may take theform of a computer program product embodied in one or more computerreadable medium(s) having computer readable program code embodiedthereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium would include the following: a portable computerdiskette, a hard disk, a random access memory (RAM), a read-only memory(ROM), an erasable programmable read-only memory (EPROM or Flashmemory), a portable compact disc read-only memory (CD-ROM), an opticalstorage device, a magnetic storage device, or any suitable combinationof the foregoing. In the context of this document, a computer readablestorage medium may be any tangible medium that can contain or store aprogram for use by or in connection with an instruction executionsystem, apparatus, processor, or device.

A computer readable signal medium may include a propagated data signalwith computer readable program code embodied therein, for example, inbaseband, as part of a carrier wave, an electrical connection having oneor more wires, an optical fiber, etc. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. Acomputer readable signal medium may be any computer readable medium thatis not a computer readable storage medium and that can communicate,propagate, or transport a program for use by or in connection with aninstruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmittedusing any appropriate medium, including but not limited to wireless,wireline, optical fiber cable, RF, etc., or any suitable combination ofthe foregoing.

Computer program code for carrying out operations for aspects of thepresent invention may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java, Smalltalk, C++ or the like and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages. The program code may execute entirely on theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection may be made to an externalcomputer (for example, through the Internet using an Internet ServiceProvider).

Aspects of the present invention are described below with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational steps to be performed on the computer, otherprogrammable apparatus or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions.

FIG. 1 illustrates an architecture 100, in accordance with oneembodiment. As shown in FIG. 1 , a plurality of remote networks 102 areprovided including a first remote network 104 and a second remotenetwork 106. A gateway 101 may be coupled between the remote networks102 and a proximate network 108. In the context of the presentarchitecture 100, the networks 104, 106 may each take any formincluding, but not limited to a LAN, a WAN such as the Internet, publicswitched telephone network (PSTN), internal telephone network, etc.

In use, the gateway 101 serves as an entrance point from the remotenetworks 102 to the proximate network 108. As such, the gateway 101 mayfunction as a router, which is capable of directing a given packet ofdata that arrives at the gateway 101, and a switch, which furnishes theactual path in and out of the gateway 101 for a given packet.

Further included is at least one data server 114 coupled to theproximate network 108, and which is accessible from the remote networks102 via the gateway 101. It should be noted that the data server(s) 114may include any type of computing device/groupware. Coupled to each dataserver 114 is a plurality of user devices 116. Such user devices 116 mayinclude a desktop computer, lap-top computer, hand-held computer,printer or any other type of logic. It should be noted that a userdevice 111 may also be directly coupled to any of the networks, in oneembodiment.

A peripheral 120 or series of peripherals 120, e.g., facsimile machines,printers, networked and/or local storage units or systems, etc., may becoupled to one or more of the networks 104, 106, 108. It should be notedthat databases and/or additional components may be utilized with, orintegrated into, any type of network element coupled to the networks104, 106, 108. In the context of the present description, a networkelement may refer to any component of a network.

According to some approaches, methods and systems described herein maybe implemented with and/or on virtual systems and/or systems whichemulate one or more other systems, such as a UNIX system which emulatesan IBM z/OS environment, a UNIX system which virtually hosts a MICROSOFTWINDOWS environment, a MICROSOFT WINDOWS system which emulates an IBMz/OS environment, etc. This virtualization and/or emulation may beenhanced through the use of VMWARE software, in some embodiments.

In more approaches, one or more networks 104, 106, 108, may represent acluster of systems commonly referred to as a “cloud.” In cloudcomputing, shared resources, such as processing power, peripherals,software, data, servers, etc., are provided to any system in the cloudin an on-demand relationship, thereby allowing access and distributionof services across many computing systems. Cloud computing typicallyinvolves an Internet connection between the systems operating in thecloud, but other techniques of connecting the systems may also be used.

FIG. 2 shows a representative hardware environment associated with auser device 116 and/or server 114 of FIG. 1 , in accordance with oneembodiment. Such figure illustrates a typical hardware configuration ofa workstation having a central processing unit 210, such as amicroprocessor, and a number of other units interconnected via a systembus 212.

The workstation shown in FIG. 2 includes a Random Access Memory (RAM)214, Read Only Memory (ROM) 216, an I/O adapter 218 for connectingperipheral devices such as disk storage units 220 to the bus 212, a userinterface adapter 222 for connecting a keyboard 224, a mouse 226, aspeaker 228, a microphone 232, and/or other user interface devices suchas a touch screen and a digital camera (not shown) to the bus 212,communication adapter 234 for connecting the workstation to acommunication network 235 (e.g., a data processing network) and adisplay adapter 236 for connecting the bus 212 to a display device 238.

The workstation may have resident thereon an operating system such asthe Microsoft Windows® Operating System (OS), a MAC OS, a UNIX OS, etc.It will be appreciated that a preferred embodiment may also beimplemented on platforms and operating systems other than thosementioned. A preferred embodiment may be written using JAVA, XML, C,and/or C++ language, or other programming languages, along with anobject oriented programming methodology. Object oriented programming(OOP), which has become increasingly used to develop complexapplications, may be used.

An application may be installed on the mobile device, e.g., stored in anonvolatile memory of the device. In one approach, the applicationincludes instructions to perform processing of an image on the mobiledevice. In another approach, the application includes instructions tosend the image to a remote server such as a network server. In yetanother approach, the application may include instructions to decidewhether to perform some or all processing on the mobile device and/orsend the image to the remote site.

In various embodiments, the presently disclosed methods, systems and/orcomputer program products may utilize and/or include any of thefunctionalities disclosed in related U.S. Patents, Patent Publications,and/or Patent Applications incorporated herein by reference. Forexample, digital images suitable for processing according to thepresently disclosed algorithms may be subjected to image processingoperations, such as page detection, rectangularization, detection ofuneven illumination, illumination normalization, resolution estimation,blur detection, classification, data extraction, etc.

In more approaches, the presently disclosed methods, systems, and/orcomputer program products may be utilized with, implemented in, and/orinclude one or more user interfaces configured to facilitate performingany functionality disclosed herein and/or in the aforementioned relatedpatent applications, publications, and/or patents, such as an imageprocessing mobile application, a case management application, and/or aclassification application, in multiple embodiments.

In still more approaches, the presently disclosed systems, methodsand/or computer program products may be advantageously applied to one ormore of the use methodologies and/or scenarios disclosed in theaforementioned related patent applications, publications, and/orpatents, among others that would be appreciated by one having ordinaryskill in the art upon reading these descriptions.

It will further be appreciated that embodiments presented herein may beprovided in the form of a service deployed on behalf of a customer tooffer service on demand.

Improved Object Detection and Image Cropping

In general, the presently disclosed inventive concepts are concernedwith providing a robust object detection and image cropping solutioncapable of precise, accurate identification of edges/boundaries betweenobject(s) of interest and background as represented in digital images.Importantly, the presently disclosed inventive concepts convey theadvantage of enabling robust object detection even in the presence ofcomplex background, within low contrast and/or oversaturated images,within images exhibiting shadows, glare, etc., and regardless of whetherthe image was generated/captured using a conventional flat-bed scanner,MFP device, and the like or using a camera.

Turning now to FIGS. 3A-3H, various embodiments of images depicting anobject under circumstances that make object detection difficult areshown.

FIG. 3A is a photographic representation of a low-contrast and/oroversaturated image 300 captured using a conventional flat-bed scanner,according to one embodiment. In the embodiment of FIG. 3A, the object302 sought for detection is a business card having black text printed ona white background. Due to the lack of contrast, edges of the businesscard are not immediately apparent within the scanned image 300.Moreover, various artifacts 304, 306 appear within the image 300,creating opportunities for false positive edge identification, characteridentification, and associated challenges that may frustrate objectdetection within the image 300. Indeed, the image is sufficientlysaturated that the text appearing on the business card is degraded inthe scanned image 300.

FIG. 3B is a photographic representation of an image 310 captured usinga conventional flat-bed scanner, the image 310 having a small object 302depicted therein, and the object 302 being located in an unconventionallocation and orientation, according to one embodiment. Similar to FIG.3A, FIG. 3B depicts a business card as the object 302 sought fordetection. Also similar to FIG. 3A, the image 310 is characterized bylow contrast and/or oversaturation, but to a lesser extent than FIG. 3A.Accordingly, image 310 also includes artifacts 304, 306 that maygenerate false positive edge location predictions. However, note thepoints appearing in region 308 substantially rest along a linear pathand may indicate a location of an upper edge of the object 302. Even so,the orientation of these points within the overall image 310 does notcomport with general assumptions that an object such as a document, whenimaged using a traditional flatbed scanner, will be orientedsubstantially in “portrait” or “landscape” fashion and positioned so asto align with the upper and left borders of the scan area. As explainedfurther below, this unconventional placement may violate an essentialassumption regarding object location and frustrate gathering appropriateinformation to use in distinguishing between foreground (object) andbackground of the image 310.

Accordingly, in FIG. 3B the primary challenge for object detection isthe unconventional positioning/orientation, and relatively low size ofthe object 302 compared to the total size image of the image 310. Asdescribed in further detail elsewhere herein, objects with relativelylow area compared to total image area are difficult to detect withoutprecise knowledge regarding the location of the object within the image.Since many object detection techniques rely on determiningcharacteristics of the background and foreground, respectively, in orderto distinguish between the two, in situations represented by FIG. 3B itis difficult to ensure the characteristics gleaned as representative ofthe “background” or “foreground” truly correspond to those portions ofthe image. This is because the object 302 occupies relatively little ofthe image (and thus represents relatively little of the informationavailable in the image), and the expected location of foreground andbackground cannot be relied upon due to the unconventional positioningof object 302 within the image 310.

FIG. 3C is a photographic representation of an image 320 captured usinga camera and exhibiting glare, according to one embodiment. Glare mayhave a similar effect as oversaturation, but confined to a local regionwithin an image 320 and potentially having a greater impact on theability to distinguish between foreground and background within theglare region 322 than other regions of the image 320, even if thoseother regions are characterized by oversaturation and/or low contrast.In particular, and especially for objects having similar colorprofile/characteristics between the foreground (object) and background,glare may defeat the ability to distinguish therebetween. In addition,glare may obscure information represented by/on the object and desiredfor use in downstream workflows/applications relying on the image data.

FIG. 3D is a photographic representation of an image 330 captured usinga camera and exhibiting distortion, according to one embodiment.Distortion of this type may cause an object represented in the image 330to deviate from expected characteristics (particularly shape) and thusfrustrate the ability to detect boundaries between the object andbackground of the image. Such perspective distortions are particularlyfrustrating in situations where vertices of a polygonal object deviatefrom expected angles, e.g. where corners of a rectangular documentdeviate from 90 degrees.

In FIG. 3E, an object 302 is depicted within an image 340 having acomplex background having three different regions each with a uniquecolor profile, according to one embodiment. A first background region342 is dominated by green colors, while second background region 344 isdominated by red and yellow colors, and third background region 346 isdominated by blue and white colors. The object sought for detection 302is a check with a substantially gray/beige color profile. In situationssuch as this, distinguishing background from foreground is exceptionallydifficult because the background is not characterized by any singlecharacteristic, or even set of characteristics. Indeed, in FIG. 3E eachbackground region 342-346 is dominated by a different channel in atypical RGB color scheme, making each background region as distinct fromeach other as from object 302.

In FIG. 3F, an object 302 is depicted on a complex background includingobjects of a similar type as the object desired for detection, accordingto one embodiment. As in FIG. 3E, the object 302 is a check, but unlikeFIG. 3E, the complexity of the background of image 350 arises from theinclusion of similar background objects 352 (blank, lined sheets ofpaper) within the background. The similar color characteristics betweenthe object 302 and background objects 352, coupled with the lines 356 onthe background objects 352 frustrate distinction between the object 302and background objects 352 because the lines are highly likely togenerate false positive edge location predictions. This is because thelines 356 exhibit substantially similar characteristics as expected toindicate the location of edges/boundaries between the foreground andbackground of image 350.

Exacerbating this problem is the fact that, according to embodimentsconsistent with FIG. 3F, additional background 354 may be included inthe image 350. The additional background 354 corresponds to one or moresurfaces (e.g. table, floor) upon which the object 302 and/or backgroundobjects 352 are arranged, and has significantly different colorcharacteristics than both the object 302 and background objects 352. Assuch, transitions from the additional background 354 to backgroundobjects 352 may be projected as the location of boundaries betweenforeground (object 302) and background, rather than transitions from thebackground objects 352 and object 302, as is the desired result ofobject detection.

In FIG. 3G an object 302 is depicted on an image 360 having a complexbackground with six different regions 364 a-364 f each exhibiting aunique color profile, according to one embodiment. Problems describedabove with reference to FIG. 3E are also present when attempting todetect objects according to the embodiment of FIG. 3G, but in FIG. 3Gcertain background regions share color characteristics, in at leastportions thereof. For instance, while region 364 a is dominated by darkcolors and region 364 f dominated by bright red and yellow/orangecolors, regions 364 c, 364 d, and 364 e each are dominated bysubstantially white colors, especially in the portions of the imageadjoining each region (i.e. along the right edge of image 360 as shownin FIG. 3G. Similarly, the upper left portion of object 302 exhibitssimilar color characteristics as surrounding background of region 364 b,i.e. near the upper left corner of the object 302. Accordingly, acomplex background may arise from drastically different color profileswithin background and foreground, such as shown in FIG. 3E, and/or dueto similarities between different background regions and/or betweenbackground and foreground, in various embodiments. A “complex”background shall therefore be understood as broadly inclusive ofbackgrounds with variable textures, color profiles, including otherobjects, etc. in various permutations or combinations as describedherein, without limitation.

In FIG. 3H, an image 370 of an object 302 is depicted with portionsthereof obscured by another object, and other portions thereof obscuredby shadow, according to one embodiment. More specifically, as shown inFIG. 3H, object 302 is a driver license held in the hand (other object)of an individual capturing the image 370 thereof. As a result, lowerborder/edge of the object 302 is partially obscured by the heel of theindividual's hand, indicated by arrow 376. Moreover, along the upperboundary/edge of the object 302, fingers and shadows cast therebyobscure the location of the upper border/edge in regions indicated byarrows 374. These obscured/shadowed regions may cause false positiveedge location predictions and/or cause the edges of the object 302 tobecome apparently curved, have extra vertices, or create other obstaclesto high fidelity object detection.

The presently disclosed inventive concepts advantageously enabledetection of objects of various types under various challengingcircumstances such as represented in FIGS. 3A-3H, providing improvedability to detect objects under varied conditions and independent of thesource of the image data. Accordingly, the instantly-describedembodiments of improved image detection represent an improvement tocomputer technology via enabling a computer to perform functions(generally, pattern recognition) typically reserved for humanperformance due to computers' well-known inability or poor capacity foraccomplishing such tasks. (Consider, for example, the use of blurredand/or complex image data to distinguish humans from computers insecurity measures such as CAPTCHA® and the like).

Moreover, by following a multi-pronged approach as described below,object detection may be performed with sufficiently low computationalcost so as to enable detection of objects in real-time or near real-timeand thus facilitate detection of a wide variety of objects within videodata, without the need to manually adapt or otherwise “train” thedetection process to detect each of the various types of objects imagedin the video. The use of a multi-detector, in particular, enables thisversatile and nimble analysis while providing desirably high recallunder a wide range of scenarios. Accordingly, the presently disclosedinventive concepts also represent an improvement to the field of imageprocessing.

Multi-Detector

Again, generally, the presently disclosed inventive concepts provideadvantages with respect to object detection, even in the presence ofundesirable artifacts or challenges described herein, particularly withreference to FIGS. 3A-3H, via use of a unique multi-detector tool andapproach in which various detection algorithms each optimized toovercome one or more specific challenges presented by some or all of theartifacts described immediately above are applied, potentially usingdifferent operational settings/parameters to determine an optimumrecognition approach or combination thereof.

Recognition results are preferably obtained from using each “detector,”which should be understood as referring to a different detectionalgorithm (e.g. color-based, line segment-based, text-based, etc. asdescribed in greater detail below) or a same/similar algorithm but withdifferent operational settings/parameters. Exemplary parameters that maybe varied to generate different “detectors” generally include, but arenot limited to, specifying fonts to seek in text line-based detection,such as OCR-B font for detecting MRZ characters, or E13B for detectingMICR characters; specifying a number and/or identity of expectedcharacters in a given text line/block, specifying a number of lines oftext of a particular type included in a text block (e.g. two lines forpassports, one line for checks, etc.), specifying expected geometriccharacteristics of features (particularly characters) expected to beincluded or represented in/on an object sought for detection, specifyinga location and/or size of a corridor within which to conduct a refinedsearch for object edges, specifying an expected angle or angle rangeexisting between adjacent sides of an object, specifying an expectedminimum contrast threshold for candidate edge pixels, and/or an expectedaspect ratio of an object sought for detection, specifying a number ofGaussians to be utilized in analyzing the image data, or any combinationthereof in accordance with various embodiments of the presentlydisclosed inventive concepts.

See also the exemplary configuration file presented below in Table 1,and additional discussion of analysis parameters set forth with respectto method 500 and FIG. 5 , below, for additional and/or alternativeanalysis parameters and exemplary values therefor. It should beunderstood that the analysis parameters may be employed in the contextof any of the exemplary methods 500-900 described herein, withoutdeparting from the scope of the present disclosure.

Recognition results from each detector applied to an input image areevaluated using confidence scores. Based on the confidence scores, thebest recognition result may be retrieved, and/or multiple detectionresults may be combined in order to form a composite result. Combiningresults may help boost overall confidence, and may enable detection insituations where no single approach/algorithm is individually suitablefor obtaining results with a desired confidence level. For instance,recognition results may have confidence scores associated with variousportions of a detected object (e.g. different corners or sides of apolygon), and the various portions obtained using each detector may bepolled to determine an optimum combination having all the best portionsobtained from the complete set of detector results.

In accordance with the myriad embodiments described herein, any suitableconfidence score or combination thereof may be employed in the contextof evaluating the detection result for a given detector.

For instance, in a line segment-based detector, multiple segments whichare candidate estimates for the location of an edge of an object may beevaluated for overall fit or linear character, e.g. using a leastsquares regression or other suitable fitness measure, and a confidencescore computed for various sets of the candidate segments. The set withthe highest confidence may be chosen as the optimum detection result forthe particular edge.

In one approach, and as shown in FIG. 4A, the object is a documenthaving four boundaries defining a tetragon/quadrilateral. Line segmentsdetected by the line detection algorithm are small fragments of linesegments. As shown in FIG. 4B, the detected line segments along theright edge of the document are broken segments. In order to project thedocument edge, these segments are grouped as one line segment. In orderto group line segments located on one line, a line segments clusteringalgorithm is applied. After grouping these broken line segments, analgorithm searches for the best quadrilateral of the documents byevaluating all possible quadrilaterals that the grouped line segmentscan form. The particular line segment clustering algorithm andquadrilateral search algorithm may be any suitable algorithm known inthe art as being suitable for line segment clustering, or quadrilateralsearching, respectively.

In order to rank quadrilaterals, the number of edge pixels projectedalong the four sides of a given quadrilateral are computed. The bestquadrilateral is the one with the largest number of projected edgepixels. Note that in order to count the projected edge pixels along eachside of a quadrilateral, the pixels on a line segment needs to beprojected onto the side of a quadrilateral.

In addition, two approaches of computing the confidence scores of foursides of quadrilateral can be applied, in various embodiments. One isuse the absolute value of the number of edge pixels projected along aside. The other is use the relative value, i.e., ratio of the number ofedge pixels projected a long a side to the number of pixels on thisside. Both methods are implemented in the new prototype system. The“absolute” confidence value is used in ranking quadrilaterals,afterwards, a “relative” confidence is also evaluated, in oneembodiment.

In more embodiments, particularly for text line detection, OCRconfidence measures may be utilized, singly or in combination with otherconfidence comparisons such as a comparison of the expected location oftext within an object versus the location of the text relative to anestimated object edge/boundary location as determined by the detectoralgorithm. For instance, in one approach determining whether a baseline,midline, topline, etc. of recognized text characters is parallel (orsubstantially parallel, e.g. within about 5 degrees angle oforientation) to a closest estimated object edge/boundary is informativeas to the quality/confidence of the OCR operation. Specifically,locating a parallel edge in proximity to detected text indicates a highconfidence in the OCR result.

In more embodiments, confidence scores or measures utilized by variousdetectors may include any combination of: 1) shape confidence score suchas an aspect ratio confidence score, and/or a score reflecting proximityto an expected angle between two adjacent sides of an object sought fordetection; 2) edge strength confidence scores; 3) confidence scores forlocations of four corners as identified by different detectors (in whichcase a majority vote approach can be used to determine which detector'sresult is best); 4) expected textual information, (for instance for MRZdocuments, if a MRZ text line is found on the detected document, a highconfidence score is returned), etc. In preferred embodiments, alldetected information is used to determine the best location of an objectwithin the image. The information can be used to train a classifier, andthe output of the classifier may include a combined score from multipledetectors, as discussed in greater detail elsewhere herein. The fusedscore may be taken as a representative confidence score of themulti-detector overall performance, in one embodiment.

The various detection algorithms may advantageously be performed inseries or in parallel, and are configured for multi-threaded operationto optimize speed of computation. In particularly preferred approaches,depending on the nature of the object to be detected, and the extent ofavailable a priori knowledge regarding characteristics of the object, anoptimum series or sequence of detectors may be defined and applied toachieve the best possible extraction result. Results from one detectormay be used to improve detection by another detector, bootstrapping theoverall detection process and improving the ultimate result.

Optionally, in some approaches one or more pre-cropping algorithms maybe applied prior to executing the multi-detector, and/or as initialiteration(s) of a multi-detector workflow, in order to reduce the amountof background present in the image upon which subsequent detectorsoperate. Additionally or alternatively, different detectors may shareinformation within a multi-detector environment, such as an embodimentin which two different detectors operate on a different color space thanthe color information represented in the input image data. So as tominimize computational cost of the overall detection process, the inputimage color space may be transformed to the desired color space (e.g.RGB to CIELUV) by the first detector, and the transformed color spacemay be stored or retained in memory, e.g. a buffer, so as to be utilizedas needed by subsequent detectors without necessitating repetition ofthe transformation. This general notion of information sharing betweenindividual detectors shall be understood as extending to all aspects ofdetector invocation, parameterization, and execution. Any pre-processingresult achieved by any given detector in the multi-detector may beleveraged by other detectors without needing to repeat thepre-processing operation.

The multi-detector is also, in accordance with preferred embodiments,adaptive and intelligent. The detector may operate in an iterativefashion, operating on multiple images potentially depicting multipledifferent types of object of interest. The multi-detector may attempt torecognize a first object using each of a plurality of detectors,optionally in real- or near-real time, and may record the confidencescores associated with each detector's result. As additional image dataare presented to the multi-detector, confidence scores may beaccumulated over time to develop a profile of detector performance atany given point in the overall detection process.

To provide adaptability, the multi-detector may provide a number ofanalysis cycles to each detector for each iteration of the detectionprocess. As understood herein, “cycles” refer to individual images orframes of image data, such that one cycle equals one image or frame, butsubsequent cycles may not necessarily represent subsequent images orframes of image data. For instance a first cycle may correspond to afirst frame of image data, while a second cycle may correspond to afourth frame of image data, particularly where the input image datacomprise video data. In response to a first detector exhibiting steadilydecreasing confidence over a predetermined number of detectioniterations, and a second detector exhibiting steadily increasingconfidence over the same interval, the multi-detector may devoteadditional cycles per iteration to the second detector, take cycles awayfrom the first detector, or both, so as to influence the overall resultsdelivered by the detector as a whole.

It is important to note that the intelligent and adaptable capabilitiesof the presently disclosed inventive multi-detector approach do notrequire performing/applying all possible or available detectoralgorithms in any given iteration, nor an iterative approach, in allcases. For instance, if a priori knowledge regarding the type of objectto be detected, and optimum detector algorithm(s) to utilize for suchtypes of object, are both available, then it is possible topre-configure the multi-detector to apply the optimum detector(s) in thefirst detection attempt.

For instance, in various embodiments optimum detector algorithm(s) maybe determined based on the source of input image data. Given the uniquechallenges associated with detecting objects in images produced usingconventional flat-bed scanners/MFPs, etc. versus detecting objects inimages produced using a camera, it is possible to define an optimumdetector algorithm or set thereof for use in detecting objects accordingto the source of input image data. As one example, a detector requiringdocument corners be substantially defined by 90 degree angles mayperform better when using flat-bed scanner input image data than camerainput image data, because the flat-bed scanner input image data does notexhibit the distortions that cause corners to be distorted away from 90degrees when captured using a camera.

In more embodiments, an image may be subjected to a classificationalgorithm, such as described in U.S. Pat. No. 9,355,312, entitled“Systems And Methods For Classifying Objects In Digital Images CapturedUsing Mobile Devices,” in order to determine the best approach todetecting objects within image data. For instance, based on thedetermined classification, a particular set of detectors may bedesignated for use in the multi-detector, preferably based on knowledgethat such detectors perform well on images classified according to thedetermined classification for the image analyzed in the first instance.For example, an image classified as being of an identification documentsuch as a passport may be subsequently analyzed using a text linedetector specifically configured to detect a particular pattern of textin a particular location of the document, such as a block of textincluding three lines of MRZ characters, each line being 30 characterslong, or a block of text including two lines of MRZ characters, eachline being 44 characters long. Upon detecting the appropriate textblock, border/edge locations may be projected based on a prioriinformation about the layout of passports (or the particulardocument/object type) and detected with high confidence and lowcomputational cost by searching within a narrow window or corridor ofthe image using optimized detection parameters.

A similar approach applies using MICR characters to locateedges/boundaries of a financial document such as a check, remittance,etc. would be understood by a person having ordinary skill in the artupon reading the present descriptions. In general, a MICR-based approachmay include determining whether an image depicts any MICR characters,and if so determining the location and/or identity thereof. In responseto determining the object does not depict any MICR characters, thisapproach may abort, allowing other detectors to proceed with analysis ofthe image. Conversely, upon determining MICR characters are present, thelocation and/or identity thereof may inform the detection process as tothe type of object represented in the image data, and likelihood thatthe object represents information useful in a downstream workflow. Forexample, an indication that an object only depicts numerical MICRcharacters along a lower portion of the object is indicative of theobject being a check or other remittance document. If a downstreamworkflow relates to financial transactions, then the detector may beparameterized so as to focus more on the particular location where theMICR characters are located, and/or to focus on detecting specificcharacteristics of the MICR characters, e.g. geometric characteristicsso as to facilitate edge location prediction.

In another, similar approach, image characteristics and/or other imagefeatures other than textual information (e.g. color profile of asubregion of the image, photographs, drawings, logos, watermarks, seals,emblems, holograms, icons, etc. as would be appreciated by a personhaving ordinary skill in the art upon reading the present descriptions)may be utilized to facilitate classification of the type of objectrepresented in the image, and an appropriate detector or set ofdetectors, optionally including a preferred order of applicationthereof, may be applied to the image data to detect the object.Generally, the image may be classified according to the location of anexpected characteristic/feature/set thereof, and an appropriate detectorchosen, or parameters thereof set, in a manner designed to optimizedetecting objects of that type.

In one embodiment, the classification may be accomplished utilizing aneural network to quickly but efficiently predict the source of theinput image data, e.g. to distinguish between camera-captured andscanned images. Upon determining the source of image data, detectors maybe chosen and/or configured to best address the unique challengesarising in the context of the particular image capture device. Forinstance, detectors configured to handle images characterized byoversaturation, low contrast, unknown object location/orientation, etc.may be employed upon determining the image source is a flat-bed scanner,MFP, or the like, while detectors configured to handle distortion and/orcomplex backgrounds may be employed upon determining the image source isa camera.

Of course, a priori knowledge regarding an input image may be obtainedfrom any suitable source, and may be included with the input image inthe form of metadata associated with the image. For instance, metadatamay identify the type of device used to capture/generate the image data,information about capture conditions such as capture angle, illuminationcharacteristics, image resolution, etc. as would be understood bypersons having ordinary skill in the art and as would be useful fordetermining appropriate processing conditions for attempting to detectobjects within an image having particular characteristics.

In a similar vein, it may be advantageous to apply various detectoralgorithms, even using default parameters, according to a particularpredetermined sequence based on the type of object to be detected withinthe image data. For instance, if an object to be detected is a document,or even better a document having a structured format such as a form,license, etc., then it may be advantageous to attempt to detect textlines prior to searching the entire image for foreground/backgroundtransitions using color information or a line segment-based approach. Aswill be appreciated by persons having ordinary skill in the art,employing a text line detector first is a computationally efficientapproach because text detection is simpler, and allows a significantreduction in the search space to which the color information and/or linesegment detectors need be applied. Accordingly, by defining anappropriate sequence of detector application based on the nature of theobject sought for detection, the presently disclosed inventivemulti-detector concepts can accomplish superior results relative toconventional techniques, or indeed any single detector algorithmdescribed herein, at a minimal computational cost.

In a preferred implementation, a technique for detecting documents inimage data employs a multi-detector in optional combination withpre-cropping techniques described in further detail below. Themulti-detector utilizes at least a text line detector and a colortransition detector, but may optionally utilize a line segment detectorin combination or as an alternative to the color transition detector toanalyze the image data. Moreover, the detectors are applied in a mannerso as to minimize computational cost associated with detection whilemaximizing detection accuracy. Accordingly, the technique involvesdetermining a source of the image data, and determining an appropriatedetector sequence in response to the source of the image data.

For instance, if the source of the image data is determined to be aflat-bed scanner, MFP, or the like, the text line detector is applied asthe initial detector so as to define a narrow search window/corridorwithin which to search for document edges/boundaries using the colortransition detector and/or line segment detector. This is advantageousbecause images of smaller documents or objects produced using aconventional flat-bed scanner/MFP/etc. are more likely to exhibit only asmall portion of the total area as foreground (document), and/or for thedocument to be located in a non-standard position/orientation, such asshown in FIG. 3B. Similarly, text may be easier to locate thanedges/borders due to tendency for flat-bed scanner-based images to beoversaturated or otherwise exhibit low contrast, and edge locations maybe hypothesized based on assuming the positions thereof aresubstantially parallel, or perpendicular to the text lines, e.g. top andbottom edges of a document are generally parallel to the baseline of atext block, and sides are typically perpendicular thereto. In furthervariations, multiple text line detectors may be employed, e.g. usingdifferent parameters such as one detector configured to detect MRZcharacters, and another detector configured to detect MICR characters.

On the other hand, in response to determining the source of the imagedata is a camera, a color transition detector may be applied as thefirst detector since such detectors have been empirically determined tooutperform text line detection when the input image data depict theobject as a substantial portion (e.g. 75% total area or more) of theimage, and the input image data include color information. Indeed,multiple color transition detectors with different operating parametersmay be employed to the exclusion of other detectors, in one embodiment.

As noted above, a pre-cropping operation may be performed prior toapplying any of the detectors in the multi-detector, or as an initialphase of a multi-detector detection process, in various approaches.

Accordingly, in one embodiment, an object detection process implementinga multi-detector as described herein is represented according to method500. The method 500 may be performed in any suitable environment,including those shown in FIGS. 1-4B, among others. Of course, more orless operations than those specifically described in FIG. 5 may beincluded in method 500, as would be understood by one of skill in theart upon reading the present descriptions.

Each of the steps of the method 500 may be performed by any suitablecomponent of the operating environment. For example, in variousembodiments, the method 500 may be partially or entirely performed bycomponents of a mobile device, a backend server, or some other devicehaving one or more processors therein. The processor, e.g., processingcircuit(s), chip(s), and/or module(s) implemented in hardware and/orsoftware, and preferably having at least one hardware component may beutilized in any device to perform one or more steps of the method 500.Illustrative processors include, but are not limited to, a centralprocessing unit (CPU), an application specific integrated circuit(ASIC), a field programmable gate array (FPGA), a graphics processingunit (GPU), etc., combinations thereof, or any other suitable computingdevice known in the art.

As shown in FIG. 5 , method 500 may initiate with operation 502, wheredigital image data are received. The image data may or may not depict anobject sought for detection, but preferably do depict such an object.The image data may be received directly from a capture device, e.g. aflat-bed scanner, multifunction printer, camera of a mobile device,webcam, video camera, or any suitable capture device as known in theart. Indeed, the presently disclosed inventive techniques advantageouslyare capable of robust processing of digital image data regardless ofsource and unique challenges associated therewith. For the purposes ofmethod 500, assume the digital image data comprise a still image.

Method 500 continues with operation 504, where the digital image dataare analyzed using a plurality of detectors, such as detectors describedin further detail below. Each detector is independently configured todetect objects within digital image data according to a unique set ofanalysis parameters. Broadly, the parameters may define the type offeatures the detector seeks to locate as indicative of presence of anobject, such as color transitions, line segments, text lines, etc.However, different detectors may also be configured to seek essentiallythe same type of features, but using different parameters (e.g.parameters with different values that slacken or tighten the constraintsby which a given detector will predict/indicate the presence of anobject, or edge thereof, such as different constraints on definingadjacent characters as belonging to a same string of characters, oradjacent pixels as belonging to a same connected component, in variousillustrative embodiments). Any difference between two given detectors issufficient to consider the detectors separate analyticaltools/processes. It should be understood that the foregoing analysisparameters may be employed in the context of any of the exemplarymethods 500-900 described herein, without departing from the scope ofthe present disclosure.

In various embodiments, the detectors may include any number ofdetectors, but preferably include at least two detectors. The two (ormore) detectors are preferably configured to detect objects according tofundamentally different approaches, e.g. color transition versus textline detection, and even more preferably are selected from: a colortransition detector configured to detect transitions between abackground and a foreground of digital image data; a line segmentdetector configured to identify a plurality of line segmentscorresponding to substantially straight lines or line segments withindigital image data; and a text line detector configured to identifyblocks of text and orientation thereof within digital image data.

In several embodiments, analyzing a still image to detect objectsinvolves determining which detector “wins” in addition to the confidenceof the detection result. For instance, again with reference to stillimages as input, a text line detector specifically configured to detectobjects based on locating a line of MRZ characters is preferred. The MRZdetector, described in detail below, produces a confidence score fordetection of the MRZ characters in addition to detection of the objectoverall. Wherein the MRZ characters have been confidently found, the MRZdetector result is preferred, regardless of whether the overallconfidence (i.e. from edge confidences, etc.) for the MRZ detector issuperior to other detectors. The MRZ detector has the benefit of a“landmark” (i.e. the MRZ lines) within the object that generic detectorsdo not, so the MRZ detector confidence is adjusted upward based ontheory and empirical observation. At least partially for this reason, anMRZ detector is preferably the first detector used in a sequence of textline detectors, again as described below and according to one exemplaryembodiment of the presently described inventive concepts.

Similarly, analysis parameters that may be employed and/or modifiedduring the course of object detection as described herein may includeconfidence thresholds for different portions of the image and/or object,such as an external area confidence threshold, a middle area confidencethreshold, an internal area confidence threshold, and a minimum edgeconfidence threshold, according to one approach.

In various approaches, each detector employed for object detection mayhave associated therewith a corresponding confidence threshold, sinceconfidence scores may be calculated in different manners based on theoperation of the respective detector. In other embodiments, confidencescores may be calibrated or normalized for comparison to a singleconfidence threshold. For instance, in one exemplary embodiment a Plattscaling method can be used to calibrate confidence values of differentdetectors. For each detector, the Platt scaling method is used toestimate the posterior probability of the detected object by minimizingthe entropy of positive and negative training examples. A crossvalidation approach is used during optimization.

In another approach, calibrating the confidence scores involvesnormalizing all confidence thresholds to a same reference value, thenlinearly scaling confidence values around the threshold.

Of course, other calibration/normalization techniques may be employed inother embodiments without departing from the scope of the presentlydescribed inventive concepts.

In more embodiments, analysis parameters may relate to include a numberof analysis iterations to perform, a downscaling target size, adownscaling target aspect ratio, a background margin size, a foregroundsize ratio, a number of foreground Gaussians, a number of backgroundGaussians, an overall energy threshold, a number of iterations in whichto apply the Gaussian(s); and/or a relative area ratio, in any suitablecombination. In preferred implementations, the foregoing analysisparameters are especially pertinent to color transition-based objectdetection. It should be understood that the foregoing analysisparameters may be employed in the context of any of the exemplarymethods 500-900 described herein, without departing from the scope ofthe present disclosure.

In further embodiments, analysis parameters may designate upper, mid,and/or lower sizes and/or dimensions for downscaled images, maximumangle deviations for forming objects from line segments (which maydiffer depending on the source of the image data, i.e.scanner/MFP/camera), minimum edge or boundary length for forming objectsfrom line segments, minimum/maximum distances between line segmentssuitable for grouping into a candidate edge set, whether to utilize abuffer “corridor” in assembling edges from line segment sets, and if sothe size and/or angle of the corridor, etc. In preferred embodiments,the foregoing analysis parameters are particularly useful in the contextof line segment-based detection. It should be understood that theforegoing analysis parameters may be employed in the context of any ofthe exemplary methods 500-900 described herein, without departing fromthe scope of the present disclosure.

In still yet more embodiments, analysis parameters may indicate: whetherto perform certain portions of an overall multi-detector-based objectdetection workflow (e.g. which detector(s) to use, and in what order,whether to perform cropping, whether to perform corner detection,whether to output a downscaled version of the input image data), whichresources or types of resources (e.g. multi-core processing) to be usedin the analysis, the source of image data (e.g. scanner versus cameraversus unknown), and/or what information to output during or uponcompleting the detection process. Generally, the foregoing analysisparameters are pertinent to any implementation of a multi-detectorapproach as described herein. It should be understood that the foregoinganalysis parameters may be employed in the context of any of theexemplary methods 500-900 described herein, without departing from thescope of the present disclosure.

Generally, analysis parameters may be expressed as Boolean values (e.g.Yes/No), integer values, and/or floating point values.

In one embodiment, for example, an exemplary configuration filespecifying suitable parameters for use in a multi-detector-based objectdetection technique includes the following definitions shown in Table 1(presented in XML format, but of course other formats may be usedwithout departing from the scope of the present descriptions). Theseparameters should be understood as illustrative only, and not limitingon the scope of the particular analysis parameters or values thereof inthe context of object detection as described herein. It should beunderstood that the analysis parameters defined in the configurationfile represented in Table 1 may be employed in the context of any of theexemplary methods 500-900 described herein, without departing from thescope of the present disclosure.

TABLE 1  <?xml version=“1.0” encoding=“utf-8”?>  <Configurationname=“XXXX”>  <Section name=“MRFDetector”>  <Parmname=“UseRandomFieldsOnly” type=“bool” value=“no” />  <Parmname=“UsePageSegmentationOnly” type=“bool” value=“no” />  <Parmname=“UseCombined” type=“bool” value=“yes” />  <Parmname=“ExternalAreaConfidenceThreshold” type=“float”  value=“0.75” /> <Parm name=“MiddleAreaConfidenceThreshold” type=“float”  value=“0.95”/>  <Parm name=“InternalAreaConfidenceThreshold” type=“float” value=“0.97” />  <Parm name=“MinimumEdgeConfidenceThreshold”type=“float”  value=“0.625” />  <Section name=“RandomFields”>  <Parmname=“MobileImage” type=“bool” value=“yes” />  <Parm name=“ScannerImage”type=“bool” value=“no” />  <Parm name=“UnknownImage” type=“bool”value=“no” />  <Parm name=“IterCount” type=“int” value=“7” />  <Parmname=“AspectRatio” type=“float” value=“−1” />  <Parmname=“RelativeAspectRatioError” type=“float” value=“0.05” />  <Parmname=“DownScaleSize” type=“int” value=“120000” />  <Parmname=“ReturnScaledImage” type=“bool” value=“no” />  <Parm name=“Mode”type=“int” value=“0” />  <Parm name=“Beta” type=“float” value=“0.025” /> <Parm name=“BackgroundMargin” type=“int” value=“8” />  <Parmname=“ForgoundSizeRatio” type=“float” value=“0.2”/>  <Parmname=“NumFgdGaussians” type=“int” value=“2”/>  <Parmname=“NumBgdGaussians” type=“int” value=“4”/>  <Parmname=“EnergyThreshold” type=“float” value=“0.05”/>  <Parmname=“NumItersGaussian” type=“int” value=“7”/>  <Parmname=“RelativeAreaRatio1” type=“float” value=“0.0001”/>  <Parmname=“RelativeAreaRatio2” type=“float” value=“0.05”/>  <Parmname=“CornerDetection” type=“bool” value=“yes” />  <Parmname=“CropImage” type=“bool” value=“no” />  <Parm name=“UseMultiCores”type=“bool” value=“yes” />  <Parm name=“RandomInit” type=“bool”value=“no” />  <Parm name=“SamplingMode” type=“int” value=“1” /> </Section>  <Section name=“PageSegmentation” >  <Parm name=“Mode”type=“int” value=“0” />  <Parm name=“MobileImage” type=“bool”value=“yes” />  <Parm name=“ScannerImage” type=“bool” value=“no” /> <Parm name=“UnknownImage” type=“bool” value=“no” />  <Parmname=“SpeedMode” type=“int” value=“4” />  <Parm name=“AspectRatio”type=“float” value=“−1” />  <Parm name=“RelativeAspectRatioError”type=“float” value=“0.1” />  <Parm name=“ReturnScaledImage” type=“bool”value=“no” />  <Parm name=“DownscaleSizeLow” type=“int” value=“240000”/> <Parm name=“DownscaleSizeHigh” type=“int” value=“1254528”/>  <Parmname=“MiniLengthThreshold” type=“float” value=“0.1”/>  <Parmname=“MaxAngleDeviation” type=“int” value=“15”/>  <Parmname=“BinThreshold” type=“float” value=“0.05”/>  <Parmname=“LineGroupThreshold” type=“float” value=“1.5”/>  <Parmname=“CropImage” type=“bool” value=“no”/>  <Parm name=“UseMultiCores”type=“bool” value=“yes” />  <Parm name=“UseAngleInRanking” type=“bool”value=“yes”/>  <Parm name=“UseCorridor” type=“bool” value=“no”/>  <Parmname=“CorridorMargin” type=“float” value=“0.10”/>  <Parmname=“CorridorMaxAngleDeviation” type=“float” value=“0.75”/>  <Listname=“CorridorFourCornersXs”>  <Parm type=“float” value=“0.0”/>  <Parmtype=“float” value=“0.0”/>  <Parm type=“float” value=“0.0”/>  <Parmtype=“float” value=“0.0”/>  </List>  <List name=“CorridorFourCornersYs”> <Parm type=“float” value=“0.0”/>  <Parm type=“float” value=“0.0”/> <Parm type=“float” value=“0.0”/>  <Parm type=“float” value=“0.0”/> </List>  </Section> </Section> </Configuration>

In operation 506, method 500 involves determining a confidence score foreach of a plurality of analysis results produced by the plurality ofdetectors. Each detector produces a result comprising a prediction ofthe location of edges or boundaries between the image background and theobject, or image foreground. Given a priori knowledge regarding theexpected characteristics, e.g. shape, size, color, etc. of the objectsought for detection, any number of techniques may be implemented toevaluate confidence in the individual predictions.

In more embodiments, the confidence scores may additionally oralternatively be determined on a more granular scale—e.g. confidence maybe evaluated for different portions of an object, such as projected edgeand/or corner (vertex) locations, curvature of curved regions of anobject, color profile for a shaded region of an object, etc. as would beappreciated by a person having ordinary skill in the art upon readingthe present descriptions. For instance, as described in further detailelsewhere herein, line segments and/or curves may be evaluated for fitand an edge location projected based on the segments, with correspondingconfidence based on the measure of fit. Details regarding the detectionand evaluation of nonlinear edges and nonstandard corners (e.g. edgesand/or intersections thereof best described by second- or higher-degreepolynomials) may be found in the disclosure of U.S. Pat. No. 9,165,187,granted Oct. 20, 2015 and entitled “Systems and Methods for Mobile ImageCapture and Processing,” which is herein incorporated by reference.

Similarly, text line detection may be informed based on confidencevalues derived from character recognition such as optical characterrecognition (OCR), intelligent character recognition (ICR) or any othersuitable recognition technique that would be appreciated by a skilledartisan reading this disclosure.

A more granular approach to confidence evaluation allows advantageouscombination of analysis results for individual portions of the object,which may be combined to generate an overall detection result withhigher confidence and quality than any individual detector could achievein solitude. As such, the presently described inventive conceptsrepresent an improvement to object detection within image data based onthe use of partial results from various different detection techniquesto generate a composite result with higher confidence and quality thanpreviously capable of being achieved using individual detectionalgorithms.

Method 500 includes operation 508, in which the analysis result (oralternatively, combined portions of various analysis results) having thehighest confidence score associated therewith is selected as the optimumobject location result.

In cases where the analysis result comprises a combination of variouspartial results from different detectors, selecting the analysis resulthaving a highest confidence score among the plurality of analysisresults as an optimum object location result may additionally include:determining, for each detector, a confidence score for each of aplurality of portions of the object, wherein each portion of the objectis determined based on the analysis result; comparing the confidencescores of each of the plurality of portions of the object determinedbased on the analysis result obtained by one of the detectors tocorresponding confidence scores determined for corresponding portions ofthe object which are determined based on the analysis result obtained byat least one other of the detectors; determining, for each of theplurality of portions of the object and based on the comparison, anoptimum one of the analysis results determined by the plurality ofdetectors; and assembling the optimum object location result based onthe optimum analysis result determined for each of the plurality ofportions of the object. In one exemplary embodiment in which the objectis a document or other object having a polygonal shape, the portions forwhich confidence measures may be individually determined may includeedges and/or corners (vertices).

In more embodiments, object detection may involve applying differentdetectors specifically configured to detect one or more portions of anobject, and combining/selecting results of the various detectors forpurposes of overall object detection. Further still, the confidencescores for partial object detectors may be combined with results fromoverall object detectors to further bootstrap the confidence in theoverall detection result.

In one exemplary approach for driver licenses, credit cards, passports,or other documents having rounded corners (which often frustratedetection algorithms attempting to define edge locations based oncalculating angles at document corners), one detector could beconfigured/parameterized to focus on the detection of rounded corners,whereas a second detector could be configured/parameterized to focus ondetection of straight edges. The multi-detector then could combineconfidence scores for the corner/edge candidates of both detectors tooptimize an overall confidence function for the object.

With continuing reference to FIG. 5 , and after selecting the optimumobject location result, method 500 includes, in operation 510, either orboth of outputting based on the optimum object location result, aprojected location of one or more edges of the object to a memory; andrendering, based on the optimum object location result, a projectedlocation of the one or more edges of the object on a display, e.g. adisplay of a computer connected to a flat-bed scanner, or a display ofthe device housing the camera.

Of course, in various embodiments, method 500 may include additionaland/or alternative operations and/or features beyond those describedabove and shown in FIG. 5 , without departing from the scope of thepresent disclosures. For instance, in one approach method 500 mayinclude downscaling the received digital image data prior to analysisthereof, to reduce computational cost and/or memory footprint associatedwith performing the presently described inventive detection techniques.The downscaling may preferably maintain the aspect ratio of the inputdigital image data, but reduce the image to a resolution ofapproximately 300×400 pixels. Any suitable downscaling technique knownin the art may be used for this purpose, without departing from thescope of the inventive concepts presented herein.

Moreover, as noted above the detection techniques presented herein maybe applied to cropping of digital images so as to remove substantiallyall background therefrom, leaving only the detected object as may bedesired for downstream processing (such as data extraction) of thecropped image data. Accordingly, in one embodiment method 500 mayadditionally or alternatively include cropping the digital image data toexclude background therefrom, the cropping being based on the projectedlocation of the one or more edges of the object.

In preferred embodiments, cropping may involve removing artifacts fromthe image, particularly artifacts such as skew, warping and perspectivedistortion, etc. that cause the shape or appearance of the object todeviate from normal. For example, in the case of an object in the formof a rectangular document, removing artifacts may involve“rectangularizing” the document, e.g. using techniques as described inU.S. Pat. No. 9,165,187, granted Oct. 20, 2015 and entitled “Systems andMethods for Mobile Image Capture and Processing;” and/or U.S. Pat. No.9,208,536, granted Dec. 8, 2015 and entitled “Systems And Methods ForThree Dimensional Geometric Reconstruction Of Captured Image Data.”

For instance, in one approach cropping in a generic sense involvescutting out a rectangular area of the original image, preferably in away that the object sought for detection is completely inside thecropping rectangle. In addition to that, for rectangular objects(particularly documents), cropping may include taking the boundaries ofthe object and creating a rectangular image that consists of the object.Accordingly, cropping may be considered a process involving at least oneof two operations: first, finding the boundaries of the object, andsecond, manipulating the image either by conventional cropping, or byprojecting the rectangular object boundaries such that a rectangularimage is created that only contains the object. For the latterprojection step, techniques such as disclosed in U.S. Pat. No.9,208,536, granted Dec. 8, 2015 and entitled “Systems and Methods forThree Dimensional Geometric Reconstruction of Captured Image Data;”and/or U.S. Pat. No. 8,855,375, granted Oct. 7, 2014 and entitled“Systems and Methods for Mobile Image Capture and Processing” may beemployed, in various embodiments. In still more embodiments, though theforegoing projection techniques are preferred, a known algorithmconfigured to perform perspective transformation on digital image datacould be implemented.

In a similar vein, but directed to facilitating object detection andimproving the robustness thereof, method 500 may include a pre-croppingoperation performed prior to the analysis of the image data by thedetectors. The pre-cropping algorithm details will be described ingreater detail below in the section entitled “Pre-Cropping.” In brief,pre-cropping removes a portion of the background positioned along theouter border of the image, thus reducing the amount of background thatneed be analyzed to determine transitions/boundaries to the foreground(object) of the image this reduces noise in the background, andgenerally improves the detection result capable of being obtained usinga multi-detector approach such as exemplified by method 500 and FIG. 5 .

In order to optimize the detection process, and specifically the typesand parameterization of detectors implemented in the analysis of thedigital image data, method 500 may include a pre-analysis step in whichthe digital image data, and/or metadata associated therewith, areanalyzed to determine whether a source of the digital image data (e.g.scanner/MFP/camera/etc.) can be determined. The analysis involvesapplying a neural network to the received digital image data, and/orparsing the metadata, e.g. to determine the source thereof and configuredetectors so as to perform optimum detection of objects therein. Asnoted in the background and introductory definitions in the detaileddescription, each source of image data conveys concomitant advantagesand disadvantages. Knowing which type of capture device was used togenerate the image data received for processing facilitates performing arobust detection process using reliable parameters of the most efficientdetector(s) available to perform detection.

Accordingly, in various embodiments, the input image data may becharacterized by any number or combination of complications. Forinstance, the object depicted in the digital image data may becharacterized by one or more of: being obscured partially or wholly byglare; being obscured partially or wholly by shadow or other objectssuch as a person's hand holding an object, or other objects present inthe field of view; having portion(s) thereof missing from the image(e.g. a cut-off corner); being oversaturated; having a low contrast withrespect to a background of the digital image; including perspectivedistortion; exhibiting warping; being depicted on a complex background;and/or having an image area less than a predetermined minimum thresholdof a total area of the digital image data.

The foregoing descriptions of multi-detector embodiments and operationthereof have been provided mainly with reference to still image data oranalyzing image frames. Skilled artisans will appreciate that theseconcepts are equally applicable to processing video data upon realizingthe computational cost of the multi-detector process and constituentdetector algorithms have been optimized so as to enable real-time ornear-real time processing thereof. Accordingly, as mentioned brieflyabove, a multi-detector analysis may be applied to a video stream byassigning certain cycles within a video sequence to different detectors,and performing detection in an iterative, adaptive and intelligentmanner so as to provide robust yet flexible detection capability.

In various approaches, a video detection implementation utilizing amulti-detector may employ any number of detectors, but in one preferredembodiment three detectors are configured. A first detector isconfigured to detect MRZ characters and define a likely edge locationwindow/corridor based upon detected MRZ characters. A second detector isconfigured to detect MICR characters and define a likely edge locationwindow/corridor based upon detected MICR characters. A third detector isa color transition detector. The multi-detector is configured to executethe foregoing detectors in the order set forth above, and by defaultassigns each detector an equal number of cycles per iteration of thedetection process.

For illustrative purposes only, assume the video-based multi detector isbeing employed as part of an application for services such as Internetservice. The applicant must provide proof of identification, proof ofresidence in the service area, and remit payment for the firstinstallment of the service period. To do so, an applicant furnishes apassport as proof of identification, a utility bill with his/her namematching the passport, and a check to tender the necessary payment. Theservice provider wishes to capture digital images of all three documentsand utilize the digital images to extract necessary information andprocess the service request.

The service provider initiates a video capture operation using a mobiledevice or other device having a camera, and invokes the multi-detectorwhile hovering the camera over the documents, which may be arranged inany order, but for sake of demonstration assume are arranged in theorder of: passport, utility bill, check. Also assume the documents areplaced on a complex background, and the passport is laminated so ittends to exhibit glare when captured using a flash setting or undercertain lighting conditions.

The multi-detector is configured with three detectors arranged tooperate in the sequence noted above, MRZ→MICR→Color, and begins with adefault distribution of equal analysis cycles across all three.

Upon the passport being placed within of the camera's field of view, theMRZ detector attempts to detect MRZ characters, and successfully does sobased on the presence of MRZ characters on the passport. As a result,the MRZ detector generates a detection result characterized by 90%confidence.

The MICR detector fares less well, there being no MICR characters todetect, and ultimately returns a bounding box including part of thebackground and only a portion of the passport, with 15% confidence.

The color transition detector, being the most robust detector disclosedherein, also successfully locates the passport, but has some difficultyinterpreting the fold in the document's center, and returns a suitabledetection result, but characterized by only 75% confidence.

After this first iteration, which may occur within less than a second,the multi-detector stores the confidence values in a historical record,and dynamically adjusts the analysis profile by altering the number ofcycles dedicated to each detector based on results of the prioriteration(s). Notably, for simplicity here adjustment occurs after eachiteration, but in practice it may be advantageous to allow themulti-detector several iterations to “burn-in” and reach a more stableconfidence estimate, then analyze several iterations' worth ofhistorical information in determining whether, and if so how, to adjustthe analysis profile of upcoming iterations. In one embodiment, analysisof the historical record is performed on a periodic basis, e.g. every 5iterations, and takes into account an average (or optionally, a weightedaverage) of the previous 3-5 iterations' confidence scores for eachdetector.

Cycles dedicated to each detector may be adjusted in any suitablemanner, but preferably detectors performing relatively well are promotedand have more cycles dedicated thereto, while detectors exhibiting lessthan desirable performance are demoted and have less cycles dedicated.In a simple example, the detector with the best average performance maybe afforded an additional cycle, up to a maximum (which depends on theframerate of the video and the computational power of the deviceperforming the analysis, e.g. 10 cycles per iteration in one embodimenthaving a 30 fps framerate, or equivalently 20 cycles per iteration foranother embodiment having a 60 fps framerate). In more complexembodiments, confidence scores may be weighted according to temporalproximity to the performance analysis, with more recent scores beinggiven more weight than older ones. For instance, in one approach using 5iterations' worth of historical data, a most recent iteration's scoremay constitute 50% of the weighted average, a second most recentiteration's score 25%, third most recent iteration's score 12.5%, fourthmost recent iteration's score 5%, and oldest iteration's score 2.5%.

Most preferably, in no case is a detector ever demoted to have zerocycles, as doing so would effectively remove the detector from theanalysis thereafter, undesirably reducing adaptability of themulti-detector as a whole. Similarly, in particularly preferredembodiments after a certain number of cycles, whether a predeterminednumber of cycles, or a predetermined number of cycles without any changeto the analysis profile, the profile may be reset to a default equaldistribution of cycles across all enabled detectors. This reset functionadvantageously facilitates the detector remaining flexible, and avoidssituations in which any given detector is so heavily weighted/favored asto be the only or dominant source of detection functionality.

As noted above, while “cycle” should be understood as synonymous with“frame,” it should be appreciated that subsequent cycles of a givendetector iteration are not necessarily consecutive. For instance, a MRZdetector may take 1 frame to perform its analysis ( 1/30^(th) second), aMICR detector may take 2 frames to perform its analysis ( 1/15^(th)second), and a color transition detector may take 6 frames ( 1/10^(th)second), such that the initial iteration takes approximately ¼ second.

Referring again to the illustrative embodiment above, in response to theMRZ, MICR, and color transition detector confidences, the multi-detectormay demote the MICR detector, promote the MRZ detector, and leave thecolor transition detector unchanged. This process may repeat for severaliterations, e.g. 4-8, while the service provider holds the phone overthe passport and until an indication of proper detection (e.g.displaying a bounding box on the video feed, displaying a cropped imageof the passport, playing a sound, etc.) is provided to the serviceprovider. At that time, the service provider moves the camera so theutility bill is in view. The multi-detector, having reinforced itsallocation of cycles each cycle while the passport was in view of thecamera, is now highly adapted to passport detection.

However, since the utility bill does not include any of the MRZcharacters the multi-detector has relied upon thus far to robustlydetect the passport, the MRZ detector reports poor results with lowconfidence, e.g. 25%. Similarly, the MICR detector continues reportingpoor results with 15-20% confidence. The color transition detectorimproves, having no internal border/fold to struggle with in detectingthe utility bill, and despite the complex background exhibits a robustconfidence of 92%. In response, the multi-detector demotes the MRZdetector and promotes the color transition detector until the analysisprofile provides routine, robust detection of the utility bill. At suchtime, an image of the bill is automatically captured and an indicationof completed detection provided to the service provider.

In a manner similar to the adjustment from the passport to the utilitybill, the multi-detector applies the MRZ, MICR, and color transitiondetectors and evaluates confidence thereof. Since the check does includeMICR characters, the MICR detector exhibits a sudden increase inconfidence, and produces robust detection results with 80% or greaterconfidence. The MRZ detector remains in the poor performance zone, therebeing no MRZ characters on the check. The color transition detectorremains at robust performance in excess of 90%. In response to the MICRdetector's improved performance, and so as to maximize the total amountof information available to synthesize into a final detection result,the MICR detector is promoted while the MRZ and color transitiondetector remain substantially unchanged. While the color transitiondetector remains at high confidence and the detection results therefrommay be fully adequate in most instances, it is advantageous to retainthis information and analyze the color transition detector's results inline with those obtained from the promoted MICR detector, optionallycombining partial or complete detection results as noted hereinabove tobootstrap the overall confidence in the detection result achieved by themulti-detector as a whole. Again, upon reaching a predeterminedconfidence overall, the multi-detector triggers an auto-captureoperation and provides an indication of successful detection to theservice provider operating the camera.

In some embodiments, the auto-capture and detection process may betriggered based on determining the confidence scores of one or more ofthe detectors have reached a predetermined stability, e.g. a confidencescore has not varied by more than 5% over a span of 3 or more cycles, 5or more cycles, 1 second, 3 seconds, 5 seconds, etc. as would beappreciated by a person having ordinary skill in the art upon readingthe present disclosures.

In more embodiments, an object detection process implementing amulti-detector and specifically configured to process video data inreal-time or near real-time is represented according to method 600. Themethod 600 may be performed in any suitable environment, including thoseshown in FIGS. 1-4B, among others. Of course, more or less operationsthan those specifically described in FIG. 6 may be included in method600, as would be understood by one of skill in the art upon reading thepresent descriptions.

Each of the steps of the method 600 may be performed by any suitablecomponent of the operating environment. For example, in variousembodiments, the method 600 may be partially or entirely performed bycomponents of a mobile device, a backend server, or some other devicehaving one or more processors therein. The processor, e.g., processingcircuit(s), chip(s), and/or module(s) implemented in hardware and/orsoftware, and preferably having at least one hardware component may beutilized in any device to perform one or more steps of the method 600.Illustrative processors include, but are not limited to, a centralprocessing unit (CPU), an application specific integrated circuit(ASIC), a field programmable gate array (FPGA), a graphics processingunit (GPU), etc., combinations thereof, or any other suitable computingdevice known in the art.

As shown in FIG. 6 , method 600 may initiate with operation 602, wherean analysis profile is defined. The analysis profile defines an initialnumber of analysis cycles that will be dedicated to each of a pluralityof detectors in an upcoming object detection analysis procedure. Theinitial number of analysis cycles dedicated to each of the plurality ofdetectors is preferably an equal distribution of cycles to each of theplurality of detectors, in one approach.

According to preferred embodiments, each analysis cycle corresponds to aparticular frame of video data based on timing, but it should beunderstood that consecutive cycles do not necessarily correspond toconsecutive frames of the video data, since a given analysis beingperformed during a cycle may take an amount of time corresponding tomultiple frames to complete. Put another way, each detector is assigneda certain number of “cycles” for each iteration of the analysis (definedby the detectors assigned to analyze the video data). Each cycleinitiates with a particular frame of the video stream, but theparticular analysis performed during that cycle may take an amount oftime corresponding to multiple subsequent frames of the video streambeing captured. As such, consecutive “cycles” of the analysis may, butdo not necessarily, correspond to consecutive “frames” of the videostream. For instance, if a cycle requires 1/10^(th) of a second tocomplete (three frames, assuming a framerate of 30 fps) and initiates onframe 1, then cycle 2 will initiate on frame 4 of the video data.

More preferably, each detector is independently configured to detectobjects using a unique set of parameters. The detectors andcharacteristics thereof as employed in the context of method 600 mayinclude any combination of features, parameters, limitations, etc.described herein, in various approaches.

Method 600 continues with operation 604, where a plurality of frames ofthe video data are received. The video data may be received by aprocessor, memory, etc. of any suitable computing device, preferably adevice capable of carrying out method 600 and also the same device thatcaptured the video data. Preferably, the video data depict at least oneobject sought for detection, but in various embodiments the presentlydisclosed inventive techniques may be employed to determine when noobject is present, e.g. based on the multi-detector returning resultsindicating low confidence in all detection attempts. In the context ofmethod 600 and the exemplary implementation thereof shown in FIG. 6 ,the video data preferably depict a plurality of unique objects notcapable of facile detection using a single approach or algorithm.

In operation 606, method 600 involves analyzing the plurality of framesusing the plurality of detectors and in accordance with the analysisprofile. Analyzing the plurality of frames essentially involvesperforming/applying the respective detection algorithm and particularparameters specified by each individual detector defined/set in themulti-detector, and produces at least one analysis result for each ofthe plurality of detectors. As noted above, each detector is assigned anumber of analysis “cycles” to be performed during each iteration of theoverall operation 606. Detectors assigned multiple cycles will producemultiple analysis results, each corresponding to a different frame ofthe video stream and providing additional information regarding theconfidence/robustness of the overall result obtained by analyzing thevideo data using the particular detector. Again, in various approachesand depending on the amount of time required to apply the algorithmspecified by the particular detector, an analysis cycle may correspondto a single frame of video data, or multiple frames of video data. Assuch, consecutive analysis cycles may, but do not necessarily,correspond to consecutive frames of the video data.

As shown in FIG. 6 , method 600 also includes operation 608, in whichconfidence scores for each analysis result are determined. Confidencemay be evaluated in any suitable manner, e.g. as described above withreference to method 500 and FIG. 5 .

Based on the confidence scores determined in operation 608, in operation610 method 600 includes updating the analysis profile. The updateessentially involves adjusting the number of analysis cycles dedicatedto some or all of the configured detectors so as to improve performancewith respect to the current object in a subsequent attempt, or toprepare the detector to detect a new object with potentiallydifferent/unique characteristics. In the former case, promoting thedetector by increasing the number of analysis cycles may be appropriate,while in the latter case “resetting” the multi-detector by setting eachcycle value to or close to the initial, default configuration may behelpful.

Preferably, the adjustments made in operation 610 to the analysisprofile: (1) afford additional cycles to detectors having confidenceresults greater than or equal to a predetermined promotion confidencethreshold; and (2) reduce, but never eliminate, a number of analysiscycles for detectors having confidence results less than a predetermineddemotion confidence threshold. In one embodiment, detectors having aconfidence greater than 80 on a scale from 0 to 100 may be promoted, anddetectors having a confidence less than 50 on the same scale may bedemoted, although other values may be chosen depending on the nature ofthe analysis and inherent difficulty in detecting objects of aparticular type. For instance, more complex objects that may be lessfacile to detect may be associated with lower promotion and/or demotionconfidence thresholds.

In more approaches, detector promotion/demotion may not be athreshold-based evaluation. For instance, in one embodiment the detectorwith the highest confidence among all detectors within the givenanalysis iteration may be promoted, and the detector(s) with the lowestconfidence within the given analysis iteration may be demoted. In moreembodiments, particularly where scores are computed as a function ofhistorical performance of a detector over time (e.g. for videoanalysis), the ‘score’ may be an (optionally weighted) average of theconfidences from the detector over some previous sliding window of time.Once the scores are computed based on the historical confidences, thenumber of analysis cycles assigned to each detector is proportional toits score, in preferred approaches. That is, scores across detectors canbe viewed as a histogram, as can the analysis cycles across detectors,and those histograms will be proportional in accordance with preferredembodiments of the presently disclosed inventive concepts.

It is important not to eliminate any given detector from the overalldetection process so as to retain the flexibility to detect multipledifferent objects having potentially drastic differences in visualcharacteristics. For instance, in the exemplary workflow set forthabove, the MICR detector was essentially useless until the camera waspositioned in view of the check, but upon such positioning the MICRdetector exhibited robust performance and was able to provide asubstantial contribution to the confidence of the overall detectionresult. In similar scenarios a given detector may be suitable only fordetecting one of a plurality of object types, but may also be the onlydetector suitable for detecting such object types. As such, detectorsshould generally not be excluded from the analysis of the video data,but instead the impact thereof adjusted based on the number of cyclesdedicated to detecting an object using the given detector.

In a particularly preferred implementation of method 600, the analyzing,the determining, and the updating are all performed in an iterativefashion until the confidence scores attain stability and do not changeby, e.g. ±5% from iteration to iteration, indicating the detectionresult has achieved optimal outcome. At this time, an image of theobject may be captured (i.e. in an operation distinct from the videocapture, and/or by designating a given frame of the video data asrepresentative of the object), and displayed on a display, e.g. with abounding box/border/edge line set overlaying the edges of the detectedobject. Additionally or alternatively the image may be stored to memory,optionally along with coordinates defining the edge locations for theobject depicted in the image. The displayed/stored image may optionallybe cropped so as to remove background therefrom as part of thedisplay/storage process.

In addition, after each iteration the confidence scores for eachdetector and/or corresponding analysis result may be collected andstored in an historical archive. An historical archive allows the use ofmultiple iterations' confidence scores in making an ultimate predictionas to the location of a given object or edge thereof within the imagedata. In addition, an historical archive can facilitate the detection ofdiverse object types by enabling an intelligent “reset” function,trigger, or time period so that objects depicted early in a given videostream do not irrevocably influence the allocation of analysis cycles ina manner optimized for detecting objects of a particular type, to thedetriment of detecting other types of objects.

Accordingly, in various approaches method 600 may include storing theconfidence score determined for each detector in a historical record. Insuch instances, the updating of cycles dedicated to each detector isbased on confidence scores determined for each of the detectors in aplurality of prior iterations, e.g. a predetermined number of iterationsand/or a predetermined amount of time. In one illustrative embodiment,confidence scores for a current iteration may be analyzed in conjunctionwith confidence scores obtained for the previous 5-10 iterations. Inanother embodiment, confidence scores from a number of iterationscorresponding to passage of an amount of time ranging from about 2 toabout 5 seconds may be considered in determining how to adjust cyclesfor upcoming iterations.

Generally, combining the confidence scores comprises a simple averagingof the scores. In still more approaches, the various confidence scoresfor different iterations may be weighted and combined according to aweighted average. For instance, confidence scores may be weightedaccording to chronological proximity to a current iteration, and theweighting preferably is proportional to the chronological proximity ofthe confidence score to the current iteration. In one embodiment, a mostproximate confidence score receives a highest weight.

In one specific implementation, which is to be considered illustrativeonly and not limiting on the scope of the present disclosure, theconfidence scores represent about 50 iterations of analysis by aparticular detector, and the weights are assigned to each iterationaccording to a function C=1−(i/i_(max)), where i is the currentiteration number, and i_(max) is the total number of iterations takeninto consideration for computing the confidence score. In suchimplementations, each weight is preferably a value in a range from 0 to1.

As described above with reference to FIG. 5 and method 500, method 600involves the use of multiple detectors in unison to accomplish animproved detection result. The detectors implemented in the context ofmethod 600 may be substantially the same as those utilized forperforming method 500, e.g. color transition, text line, line segment,etc., and may be employed in a particular order and/or using particularparameters, optionally determined based on pre-analyzing the digitalvideo data using a neural network configured to identify the source ofthe video data (e.g. the type of camera, and intrinsic characteristicssuch as focal length, resolution, color scheme, etc.).

It should be appreciated that any feature or operation described hereinwith respect to the general use of a multi-detector to improve objectdetection may be combined with, integrated into, leveraged by, orutilized in any suitable manner in combination with any of the variousother features described herein, including but not limited to particulardetector algorithms, pre-cropping, etc. as described herein.

Detector Algorithms

The instant disclosure will now explore various illustrative anddefinitive aspects of detector algorithms suitable for use in amulti-detector approach as described hereinabove. It should beappreciated that the exemplary detector algorithms described herein areprovided by way of example, and are not intended to be limiting on thescope of the present disclosures. Instead, any suitable detectionalgorithm that would be appreciated by a person having ordinary skill inthe art may be utilized in the context of the general multi-detectorapproach set forth hereinabove. Preferably, such detector algorithmsshould be capable of accurately and robustly detecting objects of atleast one particular type and further are capable of being expressed interms of a confidence score so as to enable evaluation of performanceand adjustment in real-time as-needed under the circumstances of thedetection application.

The presently disclosed inventive concepts notably include the use ofmultiple “detectors” in order to provide robust object detectioncapability regardless of the source of input image data, and in thepresence of various artifacts such as glare, shadows, low-contrast,oversaturation, unexpected target object size, location, orientation,etc. within the image, and/or others described above, in any combinationor permutation. The following descriptions set forth several suchexemplary detectors, which may be embodied as software, virtualcomponents of a computer system, or any other suitable implementationthat would be appreciated by a skilled artisan reading the presentdisclosure. Moreover, multiple detectors may be utilized in a givendetection workflow, possibly including multiple detectors of differenttypes described herein, in addition or alternative to including multipledetectors of the same type, but employing different parameters to detectobjects within image data. Any suitable combination of detectorsdescribed herein may be employed without departing from the scope of theinventive concepts presented.

Color Transition-Based Detection

Various embodiments of a detector operating on the core principle ofdetecting transitions in color information within a document will now bedescribed for illustrative purposes. Skilled artisans reading thesedescriptions should appreciate the exemplary embodiments andconfigurations set forth herein are provided for demonstrative purposesonly, and should not be considered limiting on the scope of theinventive concepts presented herein. Various features disclosedindividually with respect to a given color transition detector should beunderstood as fully compatible with other features disclosed withrespect to other color transition detectors, line segment detectors,and/or text line detectors, in various embodiments.

In essence, the color transition detector may be embodied as asemi-supervised, adaptive learning algorithm that operates under twoassumptions, the second of which is optional.

The first assumption is that pixels in spatial proximity to foursides/edges of an image correspond to background (in other words, theforeground/object is depicted in the center, and surrounded bybackground pixels along the edges of the image). In one embodiment, theregion around the outer edge of the document assumed to be backgroundmay be defined by, or comprise, a corridor approximately 8 pixelswide/tall along the respective edge of the image.

The second (optional) assumption is that the object to be detectedwithin the image is represented by pixels close to the center of theimage. Notably, the object need not be constrained to the center, butideally the center of the image depicts at least a portion of theobject. In one embodiment, the “center” of the image comprises at leasta region defined by 20% of a total image height, and 20% of a totalimage width, and is centered within the image both in terms of heightand width. Note that in several embodiments this assumption need not besatisfied, and the color transition detector is fully capable ofdetecting objects not represented within the center of the digital imagedata, though centrally-located objects are preferred. See furtherdescriptions below regarding the use of inverted color values for anexemplary implementation of color transition detection where the objectis not centrally-located within the digital image data.

With these assumptions in mind, and in accordance with one exemplaryembodiment, a color (e.g. RGB) input image is received, and the inputimage is scaled down to a predetermined resolution such as 300×400pixels. Preferably, downscaling is performed in a manner that preservesthe aspect ratio of the original image input into the multi-detector.

Pixels from the outer border region (assumed to be background) aresampled and a representative color profile of the background is compiledbased on these sampled pixels. Similarly, pixels within the centralregion of the image may be sampled and used to generate a representativecolor profile of the foreground (or object, equivalently).

To account for scenarios in which the object may not necessarily berepresented in the central region of the image, the color values forpixels in the center region may be constructed as inverted color valuesof the background pixels, in one embodiment. In other words, the numberof pixels initially selected for the foreground should ideally equal thenumber of pixels initially selected for the background, and the colorvalues of the foreground pixels are set as the inverse of the backgroundpixel color values, i.e. inverted background color.

Given the assumption that the pixels close to four sides of an image arefrom the background, the center pixels may be or may not be from adocument. A statistical model is adaptively learned by minimizing thepotential energy defined in the image. The final segmentation is thesolution of the minima of the potential energy. Contour detection isapplied to a binary image, the largest contour is the edge of thedocument, and the four sides of the documents are approximated by linesderived by least squared fitting. In various embodiments, contourdetection may be performed using any known technique that a skilledartisan reading these descriptions would consider suitable in thecontext of the inventive concepts disclosed herein.

In addition to downscaling the image, a color space transformation ispreferably applied to adjust for differences in common representationsof color in digital format versus human perception of color. Preferably,the color space transformation transforms red-green-blue (RGB) colorchannel information common to most digital images to a CIELUV colorspace more representative of human perception of colors. The CIELUVtransformation produces a plurality of Luv vectors each modeled as arandom vector in the 3D CIELUV color space. In further embodiments,detectors may operate simultaneously on different color spaces toattempt object detection.

For instance, different detectors may be configured to analyze the imagedata within different color spaces. In one embodiment, a colortransition-based detector may operate best on a Luv color space, while aline segment-based detector may operate best on a HSV color space. Othercolor spaces may be employed in conjunction with these or otherdetectors, in various embodiments and without departing from the scopeof the present descriptions. Accordingly, color space may be an analysisparameter modified during the course of detection using a multi-detectorapproach as described herein and in accordance with several exemplaryembodiments.

Upon transforming the image, and having the representative sampled colorprofile information available for foreground and background, accordingto one approach applying the color transition detector involvesadaptively learning a statistical model by minimizing the potentialenergy defined in the image. The final detection result is achieved viafinding the solution of the minima of the potential energy. A binaryimage is generated based on the assignment of various pixels as eitherforeground or background as given by the solution with the minimum totalenergy across all pixels in the image, as described in greater detailbelow (individual pixel energy is defined in equation 2). For instance,in one approach all foreground pixels may be designated white, and allbackground pixels may be designated black, so as to form the binaryimage.

In addition, in preferred approaches, contour detection is applied tothe binary image. The largest contour in the binary image is defined asthe edge of the document, and the sides of the documents areapproximated by lines derived by least squared fitting. A detectionconfidence score is computed with edge strength and shape information.In a particularly preferred embodiment, the data generation forforeground or background samples is dictated by application of multipleGaussians. A Gaussian density f(x) is defined as follows, where x is a3D color Luv vector, μ is the mean vector and Σ is a covariance matrix.Each pixel p has a color Luv value, which defines a map from p to x, themap is defined as color (p)=x or shortly denoted as x_(p):f(x)=|2πΣ^(−1/2) exp(−½(x−μ)′Σ⁻¹(x−μ))  Eqn. 1:

The number of Gaussians generally depends on the number clusters in theforeground or background images, and ideally is balanced against theacceptable runtime for the analysis, as the number of Gaussians employedis a significant factor in overall computational cost of the colortransition-based detection approach. In one embodiment, a Gaussian formodeling foreground pixels is denoted as f(x) and a second Gaussian formodeling background pixels is denoted as g(x), which takes the same formas f(x), but employs different values for the statistical parameters Σand μ. A color vector x at each pixel in foreground image is generatedby one of Gaussians, f(x), while a color vector at each pixel inbackground image is generated by one of Gaussians, g(x). In accordancewith this embodiment, the potential energy at each pixel consists of twoparts: one is the negative log likelihood of a Gaussian, the other is aninteraction energy β providing contextual information of neighboringpixels, which describes energy between two adjacent particles (here,pixels). In one embodiment, the contextual information is or correspondsto the color information of neighboring pixels, such as immediatelyadjacent upper, bottom, left and right neighbor pixels.

The local potential energy at each pixel is defined as:

$\begin{matrix}{{{LocalEnergy}\left( x_{p} \right)} = {{{Singleton}\;{{Energy}\left( x_{p} \right)}} + {\sum\limits_{q \in {{Neigbor}{(p)}}}{{DoubletonEnergy}\left( {x_{p},x_{q}} \right)}}}} & {{Eqn}\mspace{14mu} 2}\end{matrix}$

Moreover, the singleton energy and doubleton energy are defined as:SingletonEnergy(x _(p))=−log(PDF(x _(p))),  Eqn. 2A.where PDF(x) is a probability distribution function given by f(x) org(x); andDoubletonEnergy(x,y)  Eqn. 2B.is defined as −β if the y and x values are from the same Gaussianprobability density function (PDF), otherwise, the doubleton energyvalue for (x,y) is β, β describes the interaction energy between twoadjacent pixels in terms of a probability of transforming from one colorcluster to another when moving from a given pixel to a neighboringadjacent pixel. Accordingly, β acts as a penalty value for anytransition between two different color clusters exhibited by neighboringpixels. Accordingly, transitions between color clusters occur only ifthere are significant color differences between two adjacent pixels, dueto the influence of β. β is assigned a negative value to describe twoadjacent pixels without any such color cluster transition.

Those having ordinary skill in the art will appreciate that, in variousembodiments, PDF(x) is a simplified notation referring to a functionselected from among a plurality of probability distribution functions.Preferably, PDF(x) is chosen from among a plurality of Gaussians eachdescribing representative foreground or background color within thedigital image. Accordingly, for different pixels x, the function PDF(x)may vary without departing from the scope of the presently describedinventive concepts. The particular function used for modeling a givenpixel x may depend on a final clustering result achieved by an iterativeclustering procedure.

In preferred embodiments consistent with the foregoing equations, thedoubleton energy represents constraints of a state transition from aforeground pixel to a background pixel. Doubleton energy may also beexpressed as log value of such a state transition probability. Inparticularly preferred embodiments, there are two states: one forforeground, the other is for background. However, skilled artisans willappreciate that additional states and energy functions may be employedwithout departing from the scope of the present descriptions. Forexample, in order to distinguish between different types of objectssimultaneously present in an image or video stream, one state maycorrespond to background, a second state may correspond to objects of anundesired type (e.g. an object not sought for detection, such as a face,or particular document type other than a sought document type, such asthe lined pages shown in FIG. 3F), and a third state may correspond toobjects of a desired type (e.g. the check 302 shown in FIG. 3F).

Of course, other forms of singleton and/or doubleton energy functionscan also be defined and implemented without departing from the scope ofthe presently disclosed inventive color transition detectors.

As described in further detail below, the color transition detectionprocess includes minimizing the total energy of singleton and doubletonfunctions given the color data derived from the image. Minimizing theenergy in such embodiments is equivalent to maximizing the loglikelihood of the Gaussian statistical model that was used to generatethe image. The model with the optimal parameters is best fitted to theimage in terms of maximum likelihood.

The pixel neighborhood employed in conjunction with the foregoingdefinitions includes all adjacent pixels in each cardinal direction(i.e. immediately adjacent upper, lower, left and right pixels) but notdiagonally adjacent neighbors. In alternative approaches, other“neighborhoods” may be employed, such as a neighborhood including onlydiagonally adjacent pixels, or a neighborhood including cardinally anddiagonally adjacent pixels. In one approach, a neighborhood includingonly cardinally adjacent pixels is preferred for reasons ofcomputational efficiency. In other approaches, neighborhoods includingas many pixels as possible (i.e. cardinally and diagonally adjacent) ispreferred for maximum data sampling and confidence in the detectionresult.

As referred to herein, the center is pixel p, and the others are pixelq. The normalization term Z which makes the local probability summationequal 1 is ignored in the above equation. Z is the summation of exp(−LocalEnergy(x_(p))) for all pixels x_(p) in the image.

In various embodiments, a subset of pixels are initially labeled bydefault, optionally according to a binary system such that if the labelfor foreground pixels is 1, the label for background pixels is 0. Givenan image, and partial labeling information along the four outer sides ofan image as background, and/or the labeling of pixels near the center ofthe image as foreground, the image segmentation operatively identifiesthe labels for other un-labeled pixels in a manner substantially asfollows.

In preferred approaches, the label learning is a semi-supervised,adaptive, and non-linear process. The objective function is to minimizethe total potential energy across the image, which is defined as the sumof all local potential energies defined in Equations 1-2B, above.

The learning process is “semi-supervised” in situations where the onlyinformation about labeled data is the pixels close the border, e.g. forscenarios in which a document is not located at or near a central region(e.g. a region occupying about 20% of a central most area) of the image.Since this breaks the convenient assumption that the object is withinthe central region, foreground labeled data is not available.

The adaptive nature of the label learning process refers to thealgorithm's ability to identify unlabeled data, and constantly updatethe labeled data set for learning.

The label learning process involves selecting or estimating an initialGaussian model of foreground and/or background pixels from labeled data.For un-labeled pixels, the likelihoods of those pixels are computedusing the current iteration's foreground and background models. If acolor component vector is assigned a higher likelihood from theforeground model compared with background model, the pixel is labeled asa “likely foreground pixel”, otherwise it is labeled as “likelybackground pixel”. The model is then updated with new/modifiedparameters based on new labeled pixels, and the process repeatediteratively until the model parameters converge. Empirically,convergence to a predetermined threshold epsilon value may be obtainedafter relatively few, e.g. 4-8, iterations, making the detection processhighly efficient and capable of use in real-time on video data as input.As such, the number of iterations required to reach the thresholdepsilon value may be used as a convergence criterion, in variousembodiments.

Upon obtaining a predicted edge location based on the labeling(foreground/background) of individual pixels under the stable,convergent model, a segmentation confidence score is computed using edgestrength and shape information.

In various embodiments, for instance, confidence scores may be computedbased on edge strength, angles of adjacent sides of a polygon, anglesbetween opposite sides of a polygon, and/or color/texture contrastbetween foreground and background of an image.

First, regarding edge strength confidence, this measure is applicable insituations where edges of the object are not necessarily straight lines,and may even follow an irregular path such as a stochastic zig-zagpattern. In such cases, a sum of the pixels forming the edge arecounted, and a ratio between this number of edge pixels and a totalnumber of pixels included in a bounding box calculated surrounding theedge forms the confidence score. The closer the ratio is to unity, thehigher the confidence in the particular edge.

Similarly, angle confidences may be calculated based on deviations froman expected angle formed between two adjacent, or even opposite, sidesof polygon. In the simple case of a tetragon corresponding to arectangular object, angle confidence may be measured in accordance to adeviation from right angles between adjacent sides of the tetragon. Ifthe angles between adjacent sides deviate from 90 degrees, but do so ina manner that is a valid three-dimensional projection of a rectangle(e.g. as shown in FIG. 3D), then the confidence in the predictedtetragon is high.

For instance, in embodiments operating on input image data from atraditional scanner, MFP, or the like where perspective distortions andwarping effects are not expected to be present, the tolerable anglerange between two side is in a range from about 85 degrees to about 95degrees, preferably a range from about 87.5 degrees to about 92.5degrees, and most preferably in a range from about 88 degrees to about92 degrees, i.e., 2 degree variation.

On the other hand, for embodiments where input image data were capturedusing a camera, and thus perspective distortion and/or warping effectsare expected, the tolerable angle range is preferably in a range fromabout 60 degrees to about 120 degrees, more preferably in a range fromabout 70 degrees to about 110 degrees, and most preferably in a rangefrom about 75 degrees to about 105 degrees.

In still more embodiments, angles between two opposite corners of apolygon, preferably a tetragon, may be evaluated and a sum thereofcalculated. Tolerable sums of opposite angles of a tetragon are in arange from about 150 degrees to about 210 degrees, preferably a rangefrom about 160 degrees to about 200 degrees, and most preferably in arange from about 165 degrees to about 195 degrees.

It should be understood that each of the foregoing ranges is describedas an inclusive range, such that values falling on the endpoints of theranges are considered within the range.

Upon evaluating the appropriate angles, a confidence value for thecorner(s) is assigned a value of 1 (on a 0.0-1.0 scale or a 0-100 scale,equivalently) if an angle (or sum thereof) is in within the statedtolerable range, otherwise the confidence of the corner is assigned avalue of zero.

With respect to computation and use of confidence scores based oncontrast between the image foreground (detected object) and background,for objects exhibiting a substantially uniform color or images having asubstantially uniform background color profile, the contrast offoreground and background may be determined by comparing the differencebetween foreground color and background color one or more in colorspaces (e.g. RGB, CIELUV, CMYK, etc.).

However for objects and image backgrounds with complex texture orcolors, it is more difficult to measure the contrast. In one approach, apixel located proximate to a projected object edge location, e.g.,within a corridor or a region close to the object, is selected andcompared against the contrast of the object portion of the image and thebackground portion of the image within the corridor/region close to theobject.

For instance, in one embodiment the confidence score may be based oncolor or texture changes from foreground to background pixels. The firsttask is to identify locations of transitions from foreground andbackground. The statistical model learned from the image may be used todetect the locations, as described hereinabove with reference to colortransition detection. The features for the statistical model consist ofcolor and texture information.

When the locations are identified, the candidate edges of the object aresubjected to a fitness analysis, e.g. for a rectangular object thecandidate edges are fitted by a regression algorithm such as a leastmean squares (LMS) approach. For each side, a corridor is formed. Onesimple way to compute the confidence for this side is to count how manyforeground pixels are present in this corridor. In this approach, thecandidate edge pixels are labeled by the statistical segmentation modelwhich uses color and texture information, in one approach.

Another option is to build a new statistical segmentation model whichuses image data in the defined corridor to refine the originalsegmentation model. Of course, skilled artisans reading the presentdescriptions will appreciate the foregoing are only exemplary approachesto compute the confidence score based on image contrast in color ortextured images/image data.

Under varied illumination (e.g. oversaturation/undersaturation) and/orlow contrast conditions, estimating the contrast without knowing thelocation of the object within the image data is a particular challenge.One indirect way to measure the contrast is to evaluate edge confidencescores, where a higher edge confidence score value indicates a highcontrast between the background and foreground.

Accordingly, commensurate with the foregoing general descriptions ofcolor transition-based detection, and in accordance with severalillustrative embodiments of the presently disclosed inventive concepts,exemplary approaches to color transition-based detection are shown inFIG. 7 , and described below with reference to method 700. The method700 may be performed in any suitable environment, including those shownin FIGS. 1-4B, among others. Of course, more or less operations thanthose specifically described in FIG. 7 may be included in method 700, aswould be understood by one of skill in the art upon reading the presentdescriptions.

Each of the steps of the method 700 may be performed by any suitablecomponent of the operating environment. For example, in variousembodiments, the method 700 may be partially or entirely performed bycomponents of a mobile device, a backend server, or some other devicehaving one or more processors therein. The processor, e.g., processingcircuit(s), chip(s), and/or module(s) implemented in hardware and/orsoftware, and preferably having at least one hardware component may beutilized in any device to perform one or more steps of the method 700.Illustrative processors include, but are not limited to, a centralprocessing unit (CPU), a graphics processing unit (GPU), an applicationspecific integrated circuit (ASIC), a field programmable gate array(FPGA), etc., combinations thereof, or any other suitable computingdevice known in the art.

As shown in FIG. 7 , method 700 may initiate with operation 702, wheredigital image data are received. The image data may or may not depict anobject sought for detection, but preferably do depict such an object.The image data may be received directly from a capture device, e.g. aflat-bed scanner, multifunction printer, camera of a mobile device,webcam, video camera, or any suitable capture device as known in theart. Indeed, the presently disclosed inventive techniques advantageouslyare capable of robust processing of digital image data regardless ofsource and unique challenges associated therewith.

Method 700 further includes operation 704, in which the digital imagedata are analyzed using one or more color transition detectors. Eachdetector is independently configured to detect objects within digitalimage data according to a unique set of analysis parameters. Broadly,the parameters may define the type of features the detector seeks tolocate as indicative of presence of an object, such as colortransitions, line segments, text lines, etc. However, differentdetectors may also be configured to seek essentially the same type offeatures, but using different parameters (e.g. parameters with differentvalues that slacken or tighten the constraints by which a given detectorwill predict/indicate the presence of an object, or edge thereof, suchas different constraints on defining adjacent characters as belonging toa same string of characters, or adjacent pixels as belonging to a sameconnected component, in various illustrative embodiments). Anydifference between two given detectors is sufficient to consider thedetectors separate analytical tools/processes. It should be understoodthat any of the analysis parameters described herein may be employed inthe context of methods 700, without departing from the scope of thepresent disclosure.

In preferred approaches, the analysis parameters employed in the contextof method 700 include one or more parameters selected from: a number ofanalysis iterations to perform, a downscaling target size, a downscalingtarget aspect ratio, a background margin size, a foreground size ratio,a number of foreground Gaussians, a number of background Gaussians, anoverall energy threshold, a number of iterations in which to apply theGaussian(s); and/or a relative area ratio, in any suitable combination.

Of course, method 700 may include one or more of selecting andparameterizing the one or more detectors based at least in part on adetermined source of the digital image data. For instance, parametersknown to work best on scanned image data may be selected, or parametersknown to work best on camera-captured image data may be selected,depending on the determined source.

In operation 706, a confidence score for each of a plurality of analysisresults produced by the one or more color transition detectors isdetermined. The confidence score may be computed using any techniquedescribed herein, and may include computing confidence scores for theanalysis result(s) individually, in combination, or even computingconfidence scores for only portions of a given analysis result (e.g.scores for each edge, corner, etc. which may be combined to produce anoverall higher confidence result, as detailed hereinabove).

With continuing reference to FIG. 7 , operation 708 of method 700involves selecting the analysis result having a highest confidence scoreamong the plurality of analysis results as an optimum object locationresult. As noted above with regard to operation 706 and computingconfidence scores, in some approaches determining the result with thehighest confidence score may involve combining individual confidencescores for portions of the digital image data, or even optionallycombining multiple analysis results to achieve a higher confidence scorefor the overall detection result.

Upon selecting the optimum object location result, in operation 710 theresult is used to output to memory, and/or render on an appropriatedisplay, a projected location of the one or more edges of the object.This projection may take the form of 2D pixel coordinates of cornersand/or edge pixels within the digital image data, a bounding boxdisplayed in a particular color on the display, etc. as would beappreciated by persons having ordinary skill in the art upon reading thepresent descriptions.

Of course, in various embodiments, method 700 may include additionaland/or alternative operations and/or features beyond those describedabove and shown in FIG. 7 , without departing from the scope of thepresent disclosures.

For instance, in one approach, no portion of the object is presentwithin a central area of the digital image, where the central areacomprises approximately 20% of a total area of the digital imagesurrounding a central-most pixel of the digital image. In this manner,embodiments of the presently disclosed inventive detection algorithmsovercome a common problem associated with conventional detectiontechniques which rely on the assumption that the object sought fordetection is in the central area of the image (i.e. is the “focus” ofthe image).

In more approaches, method 700 may include cropping the digital imagedata to exclude background therefrom, the cropping being based on theprojected location of the one or more edges of the object. Cropping, inthe context of method 700, may include excising a rectangular portion ofthe received image data that depicts only the object sought fordetection, equivalently removing portions of the received image dataoutside the projected location of the one or more edges of the object,and/or transforming the portion of the image data within the projectedlocation of the one or more edges of the object so as to fit apredetermined shape, such as a rectangle. Transformations may generallyinclude perspective transform techniques such as described andreferenced hereinabove, in various approaches.

In particularly preferred approaches, method 700 may further include,e.g. as part of the analysis, utilizing one or more text line detectorsto generate an initial prediction of the projected location of the oneor more edges of the object. This text-line based detection is morecomputationally efficient than analyzing an entire image using a colortransition detector, but performs less well in terms of accurately andprecisely identifying object edge locations. However, using text-linedetection as an initial pass to define/refine the search area withinwhich to apply a color transition detector can significantly reducecomputational cost of detection overall, while also increasing theprecision and accuracy of edge detection. Accordingly, analyzing thedigital image data using the one or more color transition detectors maybe performed within one or more regions of the digital image datadefined by the initial prediction accomplished using the one or moretext line detectors, the one or more regions excluding at least aportion of the digital image data.

In a similar vein, and optionally in combination with text-linedetection, in several approaches method 700 may include applying one ormore pre-cropping algorithms to the digital image data prior toanalyzing the digital image data. Applying the one or more pre-croppingalgorithms effectively reduces background noise represented in thedigital image data by excluding outer border regions of the digitalimage data from the analysis. Accordingly, this pre-cropping may alsoserve to refine the search area, while improving the computationalefficiency and quality of edge detection overall.

In some approaches of method 700 analyzing the digital image may includetransforming the digital image data from a native color space to asecond color space, or multiple other color spaces. As referencedhereinabove, multiple color spaces may be analyzed simultaneously toimprove robustness of the overall detection result, and/or differentcolor spaces may be implemented in conjunction with different detectorsand/or based on the source of the image data to optimize themulti-detector overall performance.

As described in greater detail above, in preferred embodiments method700 includes, as part of the analysis of the digital image data,adaptively learning a statistical model descriptive of color transitionsbetween adjacent pixels in the digital image data, the statistical modeldescribing the color transitions in terms of an energy function;minimizing the energy function across all pixels in the digital imagedata; and designating each of the pixels in the digital image data aseither a foreground pixel or a background pixel based on the minimizedenergy function.

Notably, method 700 may additionally include computing confidence scoresfor the projected edges of the object. Preferably, the confidence scoreis based at least in part on one or more of: an edge strength of each ofthe one or more edges of the object; a value of one or more anglesformed between adjacent of the one or more edges of the object; a sum oftwo angles formed by opposite corners of a polygon representing theobject; and a color contrast between a foreground of the digital imagedata and a background of the digital image data.

Of course, method 700 may include any combination of the foregoingfeatures/operations, as well as additional or alternative featuresdescribed generally with respect to a multi-detector as set forth aboveand represented in FIGS. 5-6 and 10 , particularly concerning real-timedetection of objects within video data per method 600.

In still more approaches, a method 800 of detecting objects withindigital image data based at least in part on color transitions withinthe digital image data is shown in FIG. 8 , according to one embodiment.Of course, more or less operations than those specifically described inFIG. 7 may be included in method 800, as would be understood by one ofskill in the art upon reading the present descriptions.

Each of the steps of the method 800 may be performed by any suitablecomponent of the operating environment. For example, in variousembodiments, the method 700 may be partially or entirely performed bycomponents of a mobile device, a backend server, or some other devicehaving one or more processors therein. The processor, e.g., processingcircuit(s), chip(s), and/or module(s) implemented in hardware and/orsoftware, and preferably having at least one hardware component may beutilized in any device to perform one or more steps of the method 800.Illustrative processors include, but are not limited to, a centralprocessing unit (CPU), a graphics processing unit (GPU), an applicationspecific integrated circuit (ASIC), a field programmable gate array(FPGA), etc., combinations thereof, or any other suitable computingdevice known in the art.

In accordance with the embodiment of FIG. 8 , method 800 includesoperation 802, where a digital image depicting an object is received, orcaptured. The digital image may be captured in any suitable format, andpreferably is a color image.

Operation 804 a of method 800 involves sampling color information from afirst plurality of pixels of the digital image, and optional operation804 b involves optionally sampling color information from a secondplurality of pixels of the digital image. Each of the first plurality ofpixels is located in a background region of the digital image, whileeach of the second plurality of pixels is located in a foreground regionof the digital image. As described in greater detail above, the sampledcolor information is preferably representative of the background orforeground, respectively. However, only the background color informationis necessary, as in some embodiments color of the foreground may beestimated or inferred by inverting color of the background, essentiallydefining two zones within the image: background and “not-background”(equivalent to foreground for purposes of various embodiments theinventive concepts described herein).

With continuing reference to method 800 and FIG. 8 , operation 806includes generating or receiving a representative background colorprofile. The representative background color profile is based on thecolor information sampled from the first plurality of pixels.

Similarly, operation 808 includes generating or receiving arepresentative foreground color profile based on the color informationsampled from the second plurality of pixels and/or the color informationsampled from the first plurality of pixels. Where the representativeforeground color profile is based on the color information sampled fromthe second plurality of pixels, the representative foreground colorprofile truly represents foreground element(s) within the digital image.On the other hand, the color information from the second plurality ofpixels may not be available, in which case the representative foregroundcolor profile is based on the color information sampled from the firstplurality of pixels, and preferably in based on inverted values of thecolor information sampled from the first plurality of pixels. In thisway, the representative foreground color profile is essentially a “notbackground” region, which also serves to distinguish from backgroundpixels and is effective for locating object edges in accordance with thepresently described inventive concepts.

Each pixel within the digital image is assigned a label of eitherforeground or background using an adaptive label learning process inoperation 810. The adaptive label learning process is capable ofidentifying unlabeled data, and updating the labeled data set forfurther learning. As described in greater detail above, the adaptivelabel learning process essentially seeks to classify each pixel aseither foreground or background such that the image may be binarized andedges identified therein.

Accordingly, method 800 also includes binarizing the digital image inoperation 812. The binary value assigned to each pixel is based on therespective labels assigned to the pixels, e.g. foreground pixels may beassigned a value of one and background pixels a value of zero.

Contour detection is performed on the binarized image data in operation814, and candidate edges are evaluated from the identified contours.Preferably, a largest (longest) among the identified contours is chosenas a first edge of the object, and additional edges are sought viastatistical approximation such as least mean squares (LMS) fitting,particularly for polygonal objects, and especially four-sided objectssuch as documents. The contour detection technique invoked to identifythe contours within the binarized image may be any suitable contourdetection technique known to those having ordinary skill in the art.

In operation 816, method 800 includes defining edge(s) of the objectbased on the detected contours. As noted above, the edges may be definedbased on a result of the statistical approximation, such as LMS fitting.

In various approaches, method 800 may additionally or alternativelyinclude any combination, permutation, selection, etc. of the followingfeatures, operations, etc. without departing from the scope of theinventive concepts presented herein.

In one embodiment, method 800 may include computing a segmentationconfidence score for the defined edge(s) of the object. The segmentationconfidence score may be computed using one or more measures selectedfrom the group consisting of: edge strength, angle between adjacentedges of the object, angle between opposite edges of the object, colorcontrast between foreground and background of the image, a least meansquares fitness, and combinations thereof.

Method 800 is preferably performed on image data in which the object issurrounded by either at least 2 rows of background pixels or at least 2columns of background pixels on each side, such that the pixels fromwhich color information is sampled for determining the representativebackground color profile truly represent the background of the image.Further, where the object is located centrally within the digital image,detection thereof is simplified. Most preferably, the object occupies atleast 20% of a total area of the digital image and is locatedsubstantially in the center of the digital image.

Referring now to the adaptive label learning process invoked inoperation 810, notably the process may include selecting or estimatingat least one initial Gaussian model of the representative foregroundcolor profile and/or the representative background color profile; andperforming a maximum likelihood analysis of un-labeled pixels of thedigital image using the at least one initial Gaussian model, accordingto preferred embodiments.

The maximum likelihood analysis ideally includes minimizing a totalpotential energy across all pixels within the digital image based on therepresentative foreground color profile and the representativebackground color profile. According to such approaches, a potentialenergy of each pixel is defined by: a negative log likelihood of aGaussian model; and an interaction energy β describing a probability ofadjacent pixels exhibiting a transition from one color to another.

In one exemplary embodiment, the potential energy of each pixel may bedefined according to equation 2, above, where the SingletonEnergy(x_(p))is defined as −log (PDF(x_(p)); PDF(x) is a probability distributionfunction; and the DoubletonEnergy(x,y) is defined as either −β or β.Most preferably, the probability distribution function PDF(x) is definedas f(x)=|2πΣ|^(−1/2)exp(−½(x−μ)′Σ⁻¹(x−μ)), where x is a 3D color Luvvector, μ is a mean vector; and Σ is a covariance matrix. The values ofμ and Σ may be different for β than for −β.

Furthermore, in various approaches the adaptive learning process may beiterative, and for each iteration of the adaptive learning process, oneor more Gaussian models of the representative foreground color profileand/or the representative background color profile is/are updated basedon labels assigned to pixels in an immediately previous iteration of theadaptive learning process. For the first iteration of the process, theGaussian models may be based on the representative foreground colorprofile and/or the representative background color profile sampled fromthe digital image.

Accordingly, in one preferred approach the adaptive label learningprocess comprises: selecting or estimating at least one initial Gaussianmodel of the representative foreground color profile and/or therepresentative background color profile; and performing a maximumlikelihood analysis of un-labeled pixels of the digital image using theat least one initial Gaussian model.

Further still, the adaptive label learning process is preferablyperformed until parameters of the Gaussian model(s) achieve convergence,e.g. as measured or determined based on a predetermined thresholdepsilon convergence value. In various embodiments, the number ofiterations required to achieve the threshold epsilon value may be usedas a convergence criterion. Empirically, the inventors have discoveredthat when using a single predetermined epsilon value, the Gaussianparameters achieve convergence in about 4 to about 8 iterations of theadaptive label learning process.

An exemplary implementation of method 800 may include performing a colorspace transformation on the digital image. The color spacetransformation may involve a RGB to CIELUV transformation, or atransformation from any color space represented in the digital image toCIELUV, in various embodiments. Moreover, the color space transformationpreferably produces a plurality of Luv vectors; and each Luv vector ismodeled as a random vector in a 3D CIELUV color space.

Of course, method 800 may include any combination of the foregoingfeatures/operations, as well as additional or alternative featuresdescribed generally with respect to a multi-detector as set forth aboveand represented in FIGS. 5-6 and 10 , particularly concerning real-timedetection of objects within video data per method 600.

Line Segment-Based Detection

We turn now to detectors that operate based primarily on detection andclustering of line segments to project the location of edges/boundariesbetween an image background and the foreground or object sought fordetection. Generally speaking, the inventive line segment-baseddetection described herein is preferably implemented as a cascadedprocess. In each level of the cascade, multiple line or line segmentsare searched for within an image. Multiple images having differentresolutions may be generated and analyzed in accordance with variousembodiments of the presently disclosed line segment-based detectionalgorithms.

Generally, line segments are adaptively detected and tracked within animage. The sides of an object, e.g. for a document four sides, aredetermined by ranking all appropriate polygonal (e.g. tetragonal for adocument) candidates with confidence scores. The confidence scores aremeasured by accumulating evidence of line segments close to an edge.

More specifically, and in accordance with a preferred embodiment of linesegment detection as described herein, an input image is binarized usingmultiple binarization thresholds, at least one of which is adaptivelydetermined from the image. The adaptively determined threshold may be,or may be calculated based on, the mean and variance of the edge imageprior to applying the binarization thresholds thereto, in accordancewith one embodiment.

These thresholded images may be utilized to generate an “edge image”representing gradients derived from the original, color image, and linesegments are detected from within the edge image so as to generate a setof candidate edges based on clustering of the detected segments.

With continuing reference to the preferred embodiment of line segmentdetection, an edge image is extracted from a color image. This involvespre-processing the color image to generate amplitude values of agradient at each pixel, as well as an orientation of the gradient ateach pixel. The orientation of the gradient is represented by atwo-dimensional unit vector. Binary images are derived from the edgeimage by thresholding according to the amplitude and/or orientation ofthe gradient, i.e. in one embodiment if the amplitude value of thegradient at a pixel is above a predetermined threshold or an adaptivelydetermined threshold as described above, the pixel is labeled as acandidate edge pixel, and otherwise the pixel is labeled as a backgroundpixel. All candidate edge pixels are preferably utilized to identifyline segments within the edge image. Notably, the thresholds utilized ingenerating the binary images are in the form of angles (for gradientorientation evaluation) and amplitudes (for gradient amplitudeevaluation). In one approach, a threshold for gradient amplitude may bedetermined based on the span (or range) of the amplitude values for linesegments predicted in the edge image. The output of the binarizationprocess is a set of candidate edge pixels and respective 2-dimensionalpixel locations within the image.

Upon completing the foregoing pre-processing, line segment detectionbegins in earnest, and operates on the same gradient orientation andmagnitude information referenced above to identify the candidate edgepixels. In one particularly preferred approach, line segment detectionproceeds substantially as follows.

First, a candidate edge pixel is selected, e.g. arbitrarily, at random,based on a predetermined sequence, etc. in various embodiments and aswould be understood by a person having ordinary skill in the art uponreading the present descriptions, from among the set of candidate edgepixels to serve as an initial pixel, and set an initial line segment setusing the selected candidate edge pixel. Note that additional candidateedge pixels may be included in the line segment set in subsequentoperations and/or iterations of the line segment detection process. Forinstance, in one embodiment eight “neighbor pixels” surrounding theinitial pixel (or in subsequent iterations, current pixel) are evaluatedwith respect to gradient orientation. If a neighboring pixel hasapproximately a same gradient orientation with that of theinitial/current pixel, this neighboring pixel is added into the linesegment set, and is labeled as “visited”. This procedure is appliedrecursively, until such time as no further neighboring pixels satisfythe inclusion criterion.

Note that the measurement to see if a pixel has approximate samegradient orientation with that of the initial pixel is preferably asimilarity of those two gradients, i.e., mathematically, it is definedas the dot product of the two unit 2D vectors of those two gradients. Ifthe similarity is above a predetermined threshold, those two pixels haveapproximately the same orientation and are considered part of the sameline segment set. For instance, in one embodiment the threshold value isgiven by a function cos(t), where t is the value of the angle(s)deviation from normal, e.g. ninety degrees in the case of a rectangularobject. In one approach, a threshold value of cos(t) corresponding to anangular deviation of about 2 degrees or less may be employed,particularly for image data generated by a scanner or MFP, while in moreapproaches a threshold value of cos(t) corresponding to an angulardeviation of about 5 degrees or less may be employed, particularly forimage data generated by a camera.

Upon completion of the neighbor-clustering process, the set of pixelsclustered thereby define a “line segment set,” and pixels in the linesegment set are used to estimate the start and end points of a linesegment.

In one embodiment, identification/estimation of the start and end pointsof line segments involves applying a principal component analysis (PCA)to the line segment set, and an identifying/calculating an eigenvectorcorresponding to the maximum eigenvalue as the principal axis of theline segment, while the eigenvector corresponding to the smallesteigenvalue is taken as the normal direction of the line segment.

After the line segment direction is computed, all pixels are projectedalong this direction, and the start and end points of the line segmentare identified as the two points with the longest distance. Note thatthe line segment passes through the center of the line segment, which iscomputed as the average of those pixels in the line segment set.

To define additional line segment sets, candidate edge pixels which werenot evaluated (i.e. those not labeled “visited” during the initialrecursive search), the search algorithm is applied again with a newinitial candidate edge pixel in order to find a new line segment set.This process of identifying an initial candidate edge pixel, recursivelybuilding a line segment set therefrom based on neighbor clustering, andselection of new “initial” candidate edge pixels continues until allcandidate edge pixels are evaluated (i.e. until all pixels in thecandidate edge set are labeled “visited”).

In accordance with one exemplary embodiment of the foregoing linesegment detection procedure, an input image is represented by FIG. 4A,while the corresponding output of predicted line segments is shown inFIG. 4B. Note that the line segment prediction may be confused by thepresence of artifacts such as shadows 404, 406, as well as glare and/ortextures (folds, ridges) in the background of the image, such as shownin region 408. In FIG. 4B, a plethora of line segments that do notcorrespond to edges of the object 302 are generated based on theseartifacts. In order to minimize the impact of such artifacts, multiplebinary images may be generated using different gradient amplitude and/ororientation thresholds, and different line segment sets generated foreach image. For example, to exclude outliers, the intersect of linesegment sets across multiple binary renderings of a single input colorimage may be used as a candidate edge line segment set, in one approach.

In a similar manner, line segment-based detection may be complemented byother detection techniques which do not rely on identification of edgecandidates based on gradient analysis, such as text line detection, andthe results may be combined to accomplish a higher-confidence overalldetection result, as described in greater detail hereinabove.

With continuing reference to line segment-based detection, upongenerating a candidate edge line segment set, a polygon corresponding toan expected shape of the object sought for detection may be determinedfrom the line segments. For instance, in the case of a document thepolygon may be a quadrilateral, or tetragon. In various embodiments,two-dimensional or three-dimensional polygons may be assumed as thecorresponding shape and utilized to facilitate object detection withoutdeparting from the scope of the present disclosure. While the followingdescriptions are provided in the context of a document or otherfour-sided object, it should be understood that the general principlesmay be adapted to operate on other polygons in the context of theinventive concepts presented herein.

With continuing reference to FIGS. 4A-4B, the shape of the fourboundaries of a document is a tetragon/quadrilateral. Line segmentsdetected by the line segment detection algorithm described above formsmall fragments of overall line segments corresponding to objectboundaries, lines within the object, etc.

As shown in FIG. 4B, the detected line segments along the right boundaryof the document consist of broken segments, and should be grouped as oneline segment. In order to group line segments located on one line, aline segments clustering algorithm is applied.

While any clustering algorithm that would be considered suitable by aperson having ordinary skill in the art after reviewing the instantdisclosure may be employed without departing from the scope of theinventive concepts presented herein, in several embodiments abrute-force approach for clustering line segments may be employed. Theclustering may be based on evaluating a distance margin between adjacentsegments, a distance margin between non-adjacent segments, an overallsegment distance, or any other suitable criterion such as fitness to aregression curve, minimum segment length, etc. as would be appreciatedby a skilled artisan reading the present descriptions.

In cases where a line segment-based detector is seeking a single“proposed” line, the detector may proceed by attempting to identify allline segments close to this proposed line, for example if the minimum ofdistances of the start and end points of the line segments are within apredetermined margin threshold relative to the line, the line segment inquestion is grouped into the candidate line segment set, or line group.The process iterates until no new line segments are found. In this way,all line segments close to the line are identified as one group. For lowresolution images, the threshold may be predefined as a value of about 2pixels, while for high resolution images, the threshold may bepredefined as a value of about 8 pixels or more, in one exemplaryembodiment.

In more embodiments, given another “proposed” line, the foregoingapproach may be employed to find all line segments close to this newgiven line, and again the process may be performed iteratively for eachnew line until all suitable line segment candidate sets are identified.

A proposed line can be found/defined from among the set of all linesegment candidates, e.g. using a Hough line detection algorithm, in oneapproach.

With continuing reference to the distance margin based clustering, inone embodiment a pixel distance between an endpoint of a first segmentand a start point of a second, adjacent segment may be determined. Inresponse to determining the distance is less than a predetermineddistance threshold, e.g. 1.5 pixels, the first and second segment may bedetermined to belong to a same cluster and grouped together. Thisprocess may proceed until all appropriate adjacent clusters are grouped,and an overall edge prediction determined based thereon.

In various embodiments, the predetermined distance threshold may varydepending on the resolution of the image. For instance, for a lowresolution image (e.g. from about 256×256 to about 640×640 pixels,preferably about 490×490 pixels), a default distance threshold may be ina range from about 1.5 pixels to about 3.5 pixels, or any valuetherebetween, while for a mid-resolution image (e.g. from about 640×640pixels to about 1024×1024 pixels, preferably about 864×864 pixels) adefault distance threshold may be in a range from about 2.6 pixels toabout 6.2 pixels, and for a high resolution image (1024×1024 pixels orabove, preferably 1120×1120 pixels or above) a default distancethreshold may be in a range from about 3.4 pixels to about 8 pixels.Each of the foregoing exemplary distance threshold values and ranges setforth above were determined empirically determined using sample images.

In another embodiment, distances between two line segments (as opposedto between start and end points thereof) may be determined, and comparedto a predetermined minimum distance threshold. In response todetermining the segments have a distance less than the threshold, thesegments may be grouped into a cluster for purposes of subsequent edgeposition projection. For example, in one embodiment the distance betweentwo line segments may be measured by computing the maximum projectiondistance of all points on a first line segment to the correspondingpoints of another line segment. This distance can be derived as themaximum projection distance of the start and end point of a line segmentto the other line segment.

After grouping of broken line segments, an algorithm is employed tosearch for the best quadrilateral of the documents by evaluating allpossible quadrilaterals that the grouped line segments can form, forinstance based on calculating edge locations and/or intersectionstherebetween according to various polynomial expressions of differentdegree, e.g. as described in U.S. Pat. No. 8,855,375, granted Oct. 7,2014 and entitled “Systems and Methods for Mobile Image Capture andProcessing”. Quadrilaterals, in various embodiments, may be formed basedon identifying line segments in close proximity to a proposed line asdescribed hereinabove. A regression algorithm or other suitableequivalent thereof may be applied to the set of line segments to definethe location of the respective edge of the quadrilateral.

In order to rank quadrilaterals, the number of candidate edge pixelsprojected along the four sides of a quadrilateral are computed. The bestquadrilateral is the one whose edges include the largest number ofprojected candidate pixels determined from analyzing the binarizedimages and generating the line segment sets. In one approach, in orderto count the projected candidate edge pixels along each side of aquadrilateral, the pixels of a line segment are projected onto the sideof a quadrilateral.

In addition, two approaches of computing the confidence scores of eachside of the polygon (four sides of the quadrilateral) are preferablyapplied in concert to best evaluate the overall confidence in thepredicted polygon. In one embodiment, a first approach to evaluatingconfidence employs the absolute value of the number of candidate edgepixels projected along a given side of the polygon. In a secondapproach, the relative value, i.e., a ratio of the number of candidateedge pixels projected along a given side of the polygon to the totalnumber of pixels on the given side is employed. Both methods arepreferably implemented, and in particularly preferred embodiments the“absolute” confidence value is used in ranking quadrilaterals,afterwards, a “relative” confidence is also evaluated to “double-check”the validity of the estimation.

In particularly preferred embodiments, the foregoing line segmentdetection process is performed in a multi-pass procedure so as toincrease recall. The multi-pass procedure is implemented as a cascadedprocess.

In each stage, multiple detectors (e.g. implementations of the neighborpixel clustering using different thresholds) are employed, and if nonedetect an object of the expected characteristics, the multi-passprocedure proceeds to the next stage. In the next stage, more linesegments or line hypothesis are generated by relaxing the thresholds.This procedure is repeated until sufficient candidates are obtained toestimate an appropriate polygon.

In the particular case of enveloped objects, such as laminateddocuments, the first pass is to detection a candidate polygon. If asuitable polygon is detected, the edges of the polygon are preferablyextended into a corridor region of the image, e.g. a regioncorresponding to about 10% of a maximum detected polygon length andwidth. In the second pass, the corridor region is searched to findadditional edge candidates using the line segment detection processdescribed hereinabove.

In addition, multiple resolution images such as low resolution, middleresolution and high resolution are generated. The tetragon searchalgorithm is applied to all images in different resolutions to furtherimprove robustness and confidence of the ultimate detection result.

Accordingly, commensurate with the foregoing general descriptions ofline-segment based detection, and in accordance with severalillustrative embodiments of the presently disclosed inventive concepts,exemplary approaches to line segment-based detection are shown in FIG. 9, and described below with reference to method 900. The method 900 maybe performed in any suitable environment, including those shown in FIGS.1-4B, among others. Of course, more or less operations than thosespecifically described in FIG. 8 may be included in method 900, as wouldbe understood by one of skill in the art upon reading the presentdescriptions.

Each of the steps of the method 900 may be performed by any suitablecomponent of the operating environment. For example, in variousembodiments, the method 900 may be partially or entirely performed bycomponents of a mobile device, a backend server, or some other devicehaving one or more processors therein. The processor, e.g., processingcircuit(s), chip(s), and/or module(s) implemented in hardware and/orsoftware, and preferably having at least one hardware component may beutilized in any device to perform one or more steps of the method 900.Illustrative processors include, but are not limited to, a centralprocessing unit (CPU), a graphics processing unit (GPU), an applicationspecific integrated circuit (ASIC), a field programmable gate array(FPGA), etc., combinations thereof, or any other suitable computingdevice known in the art.

As shown in FIG. 8 , method 900 may initiate with operation 902, wheredigital image data are received. The image data may or may not depict anobject sought for detection, but preferably do depict such an object.The image data may be received directly from a capture device, e.g. aflat-bed scanner, multifunction printer, camera of a mobile device,webcam, video camera, or any suitable capture device as known in theart. Indeed, the presently disclosed inventive techniques advantageouslyare capable of robust processing of digital image data regardless ofsource and unique challenges associated therewith.

Method 900 further includes operation 904, in which the digital imagedata are analyzed using one or more line segment detectors. Eachdetector is independently configured to detect objects within digitalimage data according to a unique set of analysis parameters. Broadly,the parameters may define the type of features the detector seeks tolocate as indicative of presence of an object, such as colortransitions, line segments, text lines, etc. However, differentdetectors may also be configured to seek essentially the same type offeatures, but using different parameters (e.g. parameters with differentvalues that slacken or tighten the constraints by which a given detectorwill predict/indicate the presence of an object, or edge thereof, suchas different constraints on defining adjacent characters as belonging toa same string of characters, or adjacent pixels as belonging to a sameconnected component, in various illustrative embodiments). Anydifference between two given detectors is sufficient to consider thedetectors separate analytical tools/processes. It should be understoodthat any of the analysis parameters described herein may be employed inthe context of methods 900, without departing from the scope of thepresent disclosure.

In preferred approaches, the analysis parameters employed in the contextof method 900 include one or more parameters selected from: an uppertarget downscaled image size; a lower target downscaled image size; amiddle target downscaled image size; a camera-based maximum angledeviation for forming objects from line segments detected in the digitalimage data; a scanner-based maximum angle deviation of adjacent linesegments suitable for forming objects from the digital image data; aminimum length of line segments suitable for forming objects from thedigital image data; a maximum distance between line segments suitablefor forming objects from the digital image data; a flag indicatingwhether to compute an optional buffer corridor within which to searchfor line segments in forming objects from the digital image data; a sizeof the optional buffer corridor; and an orientation angle of alongitudinal axis of the optional buffer corridor.

Of course, method 900 may include one or more of selecting andparameterizing the one or more detectors based at least in part on adetermined source of the digital image data. For instance, parametersknown to work best on scanned image data may be selected, or parametersknown to work best on camera-captured image data may be selected,depending on the determined source.

In operation 906, a confidence score for each of a plurality of analysisresults produced by the one or more line segment detectors isdetermined. The confidence score may be computed using any techniquedescribed herein, and may include computing confidence scores for theanalysis result(s) individually, in combination, or even computingconfidence scores for only portions of a given analysis result (e.g.scores for each edge, corner, etc. which may be combined to produce anoverall higher confidence result, as detailed hereinabove).

With continuing reference to FIG. 8 , operation 908 of method 900involves selecting the analysis result having a highest confidence scoreamong the plurality of analysis results as an optimum object locationresult. As noted above with regard to operation 906 and computingconfidence scores, in some approaches determining the result with thehighest confidence score may involve combining individual confidencescores for portions of the digital image data, or even optionallycombining multiple analysis results to achieve a higher confidence scorefor the overall detection result.

Upon selecting the optimum object location result, in operation 910 theresult is used to output to memory, and/or render on an appropriatedisplay, a projected location of the one or more edges of the object.This projection may take the form of 2D pixel coordinates of cornersand/or edge pixels within the digital image data, a bounding boxdisplayed in a particular color on the display, etc. as would beappreciated by persons having ordinary skill in the art upon reading thepresent descriptions.

Of course, in various embodiments, method 900 may include additionaland/or alternative operations and/or features beyond those describedabove and shown in FIG. 8 , without departing from the scope of thepresent disclosures.

For instance, in one approach analyzing the digital image data comprisesidentifying and tracking a plurality of line segments within the digitalimage data, at least some of the line segments corresponding to theedges of the object.

Optionally, the method 900 may also include generating an edge imagefrom the digital image data, where the digital image data comprise colorinformation, and the edge image comprises a plurality of pixels eachhaving a gradient orientation and a gradient amplitude associatedtherewith.

Further still, in various embodiments, the method 900 may includegenerating a plurality of binarized images from the edge image, thegenerating being based on a plurality of binarization thresholds, atleast one of the binarization thresholds being adaptively determinedfrom the edge image, and each binarization threshold corresponding toeither or both of a predetermined gradient amplitude and a predeterminedgradient orientation.

Further still, method 900 may involve a neighbor-based recursiveclustering of pixels according to similar gradient characteristics, soas to form line segments. Such approaches preferably include: selectingan initial candidate edge pixel within the digital image data;clustering neighboring pixels having a gradient orientationsubstantially similar to a gradient orientation of the initial candidateedge pixel; and recursively repeating the clustering with furtherneighboring pixels until no neighboring pixels are characterized by agradient orientation substantially similar to the gradient orientationof the initial candidate edge pixel.

Once line segment sets are determined, e.g. using a clustering approachas described above, method 900 may employ principal component analysis(PCA) on each set, and computing an eigenvector corresponding to amaximum eigenvalue of the principal component analysis as a principalaxis of an overall line segment represented by the plurality of linesegments. In this way, line segments may be evaluated for furtherclustering and/or inclusion in a candidate edge line segment set. Thefurther clustering may involve clustering overall line segmentscharacterized by a relative distance less than a predetermined minimumdistance threshold.

In more approaches, method 900 may include cropping the digital imagedata to exclude background therefrom, the cropping being based on theprojected location of the one or more edges of the object. Cropping, inthe context of method 900, may include excising a rectangular portion ofthe received image data that depicts only the object sought fordetection, equivalently removing portions of the received image dataoutside the projected location of the one or more edges of the object,and/or transforming the portion of the image data within the projectedlocation of the one or more edges of the object so as to fit apredetermined shape, such as a rectangle. Transformations may generallyinclude perspective transform techniques such as described andreferenced hereinabove, in various approaches.

In particularly preferred approaches, method 900 may further include,e.g. as part of the analysis, utilizing one or more text line detectorsto generate an initial prediction of the projected location of the oneor more edges of the object. This text-line based detection is morecomputationally efficient than analyzing an entire image using a linesegment detector, but performs less well in terms of accurately andprecisely identifying object edge locations. However, using text-linedetection as an initial pass to define/refine the search area withinwhich to apply a line segment detector can significantly reducecomputational cost of detection overall, while also increasing theprecision and accuracy of edge detection. Accordingly, analyzing thedigital image data using the one or more line segment detectors may beperformed within one or more regions of the digital image data definedby the initial prediction accomplished using the one or more text linedetectors, the one or more regions excluding at least a portion of thedigital image data.

In a similar vein, and optionally in combination with text-linedetection, in several approaches method 900 may include applying one ormore pre-cropping algorithms to the digital image data prior toanalyzing the digital image data. Applying the one or more pre-croppingalgorithms effectively reduces background noise represented in thedigital image data by excluding outer border regions of the digitalimage data from the analysis. Accordingly, this pre-cropping may alsoserve to refine the search area, while improving the computationalefficiency and quality of edge detection overall.

Of course, method 900 may include any combination of the foregoingfeatures/operations, as well as additional or alternative featuresdescribed generally with respect to a multi-detector as set forth aboveand represented in FIGS. 5-6 and 10 , particularly concerning real-timedetection of objects within video data per method 600.

Text Line-Based Detection

Text-based detection algorithms can be a very powerful tool in overalldetection of objects of interest, particularly where a priori knowledgeis available regarding the relationship of the text to objectedges/boundaries. In such cases, it is possible to significantly narrowthe search space within which object edges/boundaries need be located.Accordingly, particularly in embodiments where the object of interest isa document having a known, predefined structure in terms of location oftext (e.g. a standard form, such as a driver license or passport, acheck, a credit card, a mortgage application, tax form, insurance claim,etc.), it may be advantageous to utilize text-based detection, e.g. as apreliminary pass in an overall detection process, so as to narrow theregion within which more complicated, e.g. color-based and/or linesegment-based detection, need be performed. Text-based detection maytherefore offer both the advantage of more accurate object detection, aswell as a more efficient process, since the color-based and/or linesegment-based analyses only need be performed within the narrowed searcharea, not on the image as a whole.

In one embodiment, text-based detection may involve/include anyfunctionality described in related U.S. Pat. No. 9,760,788, entitled“Mobile Document Detection and Orientation Based on Reference ObjectCharacteristics,” the contents of which are herein incorporated byreference.

In more embodiments, techniques such as disclosed in U.S. Pat. No.9,760,788, as well as those described herein, may be implemented using amaximum stable region approach. In brief, upon determining a region(e.g. a potential character) is a “maximum stable region” according tothe known meaning thereof in the art, text block/line locations may beestimated (e.g. based on geometric characteristics thereof) and fed intoa clustering algorithm to generate edge location candidates. Based onthe edge location candidates, line segments corresponding, e.g. to thebaseline, midline, etc. of the text blocks and/or individual charactersmay be computed and utilized to predict edge locations using aclustering approach. For instance, in one embodiment a clusteringapproach such as described hereinabove regarding line segment-baseddetection may be employed.

Pre-Cropping

As referred to generally above, in some approaches the presentlydisclosed inventive object detection processes may be implemented inconjunction with a pre-cropping procedure. Pre-cropping an image priorto input to the multi-detector described above advantageously reducesthe amount of background present in the image, bootstrapping thesampling of foreground (object) information and facilitating improvedrecall and computational efficiency in the overall detection process. Inbrief, pre-cropping involves analyzing an image to determine a roughlocation of an object, and relies on an assumption that the object islocated centrally within the image.

Accordingly, in one embodiment, a technique for pre-cropping digitalimage data so as to remove background therefrom is represented accordingto method 1000. The method 1000 may be performed in any suitableenvironment, including those shown in FIGS. 1-4B, among others. Ofcourse, more or less operations than those specifically described inFIG. 10 may be included in method 1000, as would be understood by one ofskill in the art upon reading the present descriptions.

Each of the steps of the method 1000 may be performed by any suitablecomponent of the operating environment. For example, in variousembodiments, the method 1000 may be partially or entirely performed bycomponents of a mobile device, a backend server, or some other devicehaving one or more processors therein. The processor, e.g., processingcircuit(s), chip(s), and/or module(s) implemented in hardware and/orsoftware, and preferably having at least one hardware component may beutilized in any device to perform one or more steps of the method 1000.Illustrative processors include, but are not limited to, a centralprocessing unit (CPU), an application specific integrated circuit(ASIC), a field programmable gate array (FPGA), etc., combinationsthereof, or any other suitable computing device known in the art.

As shown in FIG. 10 , method 1000 may initiate with operation 1002,where an input image is downscaled to a predetermined resolution.Downscaling advantageously reduces the amount of information present inthe image, most of which is unnecessary for the rough predictionperformed in the pre-cropping stage of the inventive detection proceduredescribed herein. In one embodiment, the image may be downscaled to apreferred resolution of about 400×300 pixels. More preferably, thedownscaling is performed in a manner that preserves the aspect ratio ofthe input image. In one approach, downscaling may be performed using apublicly-available API in the openCV library.

In operation 1004, method 1000 proceeds with application of ade-blurring algorithm to reduce the impact of photographs, signatures,and other prominent features of the image that may otherwise generate afalse positive indication of an edge of an object. Any known de-blurringtechnique may be utilized without departing from the scope of thepresently described inventive concepts. In one embodiment, de-blurringmay be implemented using a publicly-available API in the openCV library.

As shown in FIG. 10 , method 1000 also includes operation 1006, whereinthe de-blurred image is divided into a plurality of preferably squaresegments. The particular size and number of segments may be modifiedbased on the resolution of the original image, the downscaled image, orany other suitable criterion that would be appreciated by a personhaving ordinary skill in the art upon reading the present descriptions.In preferred embodiments, the downscaled image is divided into 1200equal-sized segments, forming a 40×30 grid where each segment isapproximately 10×10 pixels in size. Of course, for other downscaledresolution images, the grid may have a different number and/or size ofsegments and/or different grid dimensions. For downscaled images havinga resolution of 400×300 pixels a 10×10 pixel segment size wasempirically determined to be the best performing implementationoperating under acceptable computational cost constraints.

As noted above, pre-cropping per method 1000 assumes the object soughtfor detection is present at the center of the image. Accordingly, inoperation 1008 color distances between neighboring segments arecomputed, where the distances are color value distances between centralpixels of the neighboring segments. In other words, a color of a centralpixel of a first segment is compared against a color value of a centralpixel of a second, adjacent segment, and the difference therebetweenobtained. In various embodiments, neighboring segments may include onlythose segments immediately adjacent a given segment in the four cardinaldirections, only those segments immediately adjacent a given segment indirections diagonal from the current segment, or both.

In more approaches, the color representative of each segment may bedetermined based on sampling color information from a plurality ofpixels within the respective segment. For instance, in severalembodiments a small window or neighborhood of pixels located at or nearthe center of the segment may be sampled for color information, and thisinformation combined in an appropriate manner to determine therepresentative color for the segment.

Regarding sampling color information from various pixels to determine arepresentative color for each segment, in one exemplary approach acentral pixel and immediately adjacent pixels in cardinal directions(i.e. up, down, left and right, or north, south, east and west) may besampled, in which a total of 5 pixels contribute to the representativecolor. In another approach, a central pixel and immediately adjacentpixels in diagonal directions may be sampled, again generating arepresentative color based on information from 5 pixels. In moreembodiments, the central pixel and all cardinally and diagonallyadjacent pixels may be sampled, yielding a representative color for thesegment based on color information gleaned from 10 pixels. In still moreembodiments, a central pixel, all immediately adjacent pixels, and allpixels surrounding the immediately adjacent pixels may be sampled,yielding a representative color determined based on information from 25pixels.

The sampled color information may be combined in any suitable manner,such as by computing an average, median, mode, range, etc. of colorvalues across each channel, a weighted average of color values, etc. aswould be appreciated by a person having ordinary skill in the art uponreading the instant descriptions. For instance, in one embodimentweights may be applied based on pixel position relative to the center ofthe segment, with more central pixels being given greateremphasis/weight than surrounding pixels. In another embodiment, weightsmay be assigned based on the color channel sampled. In still moreembodiments, weights may be assigned based on expectations and/or apriori knowledge regarding the type of object sought for detection, andcolor characteristics thereof. Of course, combinations of weightingassignment schemes may be employed without departing from the scope ofthe present disclosure.

For instance, intensity values from a particular color channel orchannels may be emphasized or deemphasized based on expectationsregarding the object sought for detection. In one approach, an objectsought for detection is known to include a feature such as a marking,logo, texture, etc. in a central region thereof, the feature beingcharacterized by a particular color profile. If the color profile of thecentral feature is known to exclude (on include only minimal)contributions from a particular color channel, then that channel may bedeemphasized via appropriate weighting so as to reduce the impact ofrandom noise on the representative color calculation. Alternatively, toreduce false positive clustering, the particular color channel known tobe absent or minimally present in the central feature of the objectsought for detection may be emphasized via a higher weighting,increasing the likelihood that the distance caused by a given segmentincluding the absent/minimally present color channel contribution.

In particularly preferred embodiments, the color distances computed inoperation 1008 are based on color values transformed from an RGB spaceto a CIELUV color space, most preferably CIE94, as defined and describedin further detail at https://en.wikipedia.org/wiki/Color_difference asof Nov. 30, 2017, herein incorporated by reference.

In operation 1010, the color distances between a given segment and eachneighboring segment are compared against a predetermined threshold, andin operation 1012 segments having a distance less than the predeterminedthreshold are clustered to form a rough estimate of segments of theimage which depict the object (as opposed to the background).

Upon determining the segments believed to belong to the object(foreground), in operation 1014 a connected structure may be computedbased the shape of the segments clustered in operation 1012. Briefly,this connected structure may have any shape, but preferably conformssubstantially to the expected shape of the object to be detected in theimage.

At any rate, the shape of the object is determined and in operation 1016a polygon corresponding to the expected shape of the object sought fordetection is formed so as to bound the outer portions of the shapeformed from the connected segments. A third boundary corresponding toedges of the image is also employed to evaluate the clustering andpredict suitable cropping boundaries for the image.

In one embodiment, the bounding polygon may be defined based on applyinga Canny algorithm to the downscaled, deblurred image to estimate edgepixels within the image data. Thereafter, a Hough transform may beapplied to identify lines from the estimated edge pixels, and a polygongenerated from the identified lines. The polygon generation may beaccomplished using any suitable technique known in the art, but inpreferred embodiments generally involves attempting to form a tetragonusing each possible combination of four lines selected from the edgeimage (generated by Hough transform). For each set of lines that doesnot conform to a predetermined set of geometric parameters (e.g.specifying appropriate angles formed by intersections between adjacentlines in each set of four candidate lines, as described hereinabove),the set is omitted from further consideration (although individual lineswithin the set may be further considered in combination with differentlines to form different sets). After forming all possible tetragons, andfiltering in this manner, the remaining list of candidate tetragons areevaluated for confidence by calculating a score as the product of avalue of the tetragon square and a sum of individual line confidencesreturned by the Hough algorithm. The tetragon with the highest score isselected as the representative tetragon of the object in the image.

In preferred approaches, the value of the tetragon square is anapproximation of fitness to a square. Each corner (or computedintersection between adjacent sides) of the projected tetragon isassigned a value p₁ . . . p₄. The square of the distance between cornersp₁ and p₂ (designated p₁₂), between corners p₂ and p₃ (designated p₂₃),between corners p₃ and p₄ (designated p₃₄), and between corners p₄ andp₁ (designated p₄₁), are each computed, giving an approximation of thelength of each side of the tetragon. The square distance of oppositesides are summed, i.e. a sum of p₁₂ and p₃₄ is computed (designatedas₁), and a sum of p₂₃ and p₄₁ is computed (designated as₂). Finally,the value of the tetragon is given by the product of as₁ and as₂(designated s₁₂). The tetragon with the highest value s₁₂ is chosen asthe representative tetragon of the object in the image, in accordancewith preferred embodiments.

In operation 1018, the connected structure is compared against thebounding polygon, and if the fraction of segments included within boththe connected structure and the polygon is greater than a predeterminedthreshold, then in operation 1020 a the boundary between the object andthe image background is set based on the polygon boundaries. Forinstance, in one approach a dividend of the square length of thebounding polygon (i.e. a sum of square lengths of each side of thepolygon) and the square length of the sides of the image as a whole isideally less than a predetermined threshold of about 0.2, particularlywhere the polygon is a tetragon. Moreover, ideally the fraction ofsegments included within both the connected structure and the boundingpolygon is greated than a predetermined threshold of about 0.6, againparticularly where the polygon is a tetragon.

Otherwise, in operation 1020 b operations 1008-1016 are iterativelyrepeated with progressively higher color difference threshold valuesuntil the threshold is exceeded by the number of segments in theconnected structure. In one embodiment, increasing color differencethresholds may have values of about 1.6, 2.1, 3.1, 3.6, 4.1, 5.1, 6.6,8.6, and 11.1, each to be used in a successively more “greedy” iterationof the segment clustering process. Thresholds may proceed according to alinear step, e.g. of about 0.5, or in a nonlinear fashion, as describedin the exemplary embodiment immediately above, in accordance withvarious embodiments.

Upon achieving a polygon surrounding an object with a fraction ofsegments exceeding the predetermined threshold, the polygon boundary maybe utilized as a crop line to refine the image and remove backgroundpixels outside the polygon boundary in operation 1014. The resultingpre-cropped image may be fed to the multi-detector as input for furtherimproved object detection, according to various embodiments.

While the present descriptions of improved object detection and imagecropping have been made with primary reference to methods, one havingordinary skill in the art will appreciate that the inventive conceptsdescribed herein may be equally implemented in or as a system and/orcomputer program product.

For example, a system within the scope of the present descriptions mayinclude a processor and logic in and/or executable by the processor tocause the processor to perform steps of a method as described herein.

Similarly, a computer program product within the scope of the presentdescriptions may include a computer readable storage medium havingprogram code embodied therewith, the program code readable/executable bya processor to cause the processor to perform steps of a method asdescribed herein.

The inventive concepts disclosed herein have been presented by way ofexample to illustrate the myriad features thereof in a plurality ofillustrative scenarios, embodiments, and/or implementations. It shouldbe appreciated that the concepts generally disclosed are to beconsidered as modular, and may be implemented in any combination,permutation, or synthesis thereof. In addition, any modification,alteration, or equivalent of the presently disclosed features,functions, and concepts that would be appreciated by a person havingordinary skill in the art upon reading the instant descriptions shouldalso be considered within the scope of this disclosure.

Accordingly, one embodiment of the present invention includes all of thefeatures disclosed herein, including those shown and described inconjunction with any of the FIGS. Other embodiments include subsets ofthe features disclosed herein and/or shown and described in conjunctionwith any of the FIGS. Such features, or subsets thereof, may be combinedin any way using known techniques that would become apparent to oneskilled in the art after reading the present description.

While various embodiments have been described above, it should beunderstood that they have been presented by way of example only, and notlimitation. Thus, the breadth and scope of an embodiment of the presentinvention should not be limited by any of the above-described exemplaryembodiments, but should be defined only in accordance with the followingclaims and their equivalents.

What is claimed is:
 1. A computer-implemented method for pre-croppingdigital image data to remove background from the digital image data,comprising: dividing the digital image into a plurality of segments;computing a color value distance between corresponding pixels of one ormore pairs of neighboring segments of the digital image; comparing thecolor value distance computed for some or all of the pairs ofneighboring segments against a minimum color distance threshold;clustering all pairs of neighboring segments having a color valuedistance less than or equal to the minimum color distance threshold;computing a connected structure based on a shape of the clusteredsegments; computing a polygon bounding the connected structure;comparing a fraction of segments included in both the connectedstructure and the polygon, relative to a total number of segments in thedigital image, to a minimum included segment threshold; and in responseto determining the fraction of segments included in both the connectedstructure and the polygon, relative to the total number of segments inthe digital image meets or exceeds a minimum included segment threshold,cropping the digital image based on edges of the polygon.
 2. The methodas recited in claim 1, comprising, in response to determining thefraction of segments included in both the connected structure and thepolygon, relative to the total number of segments in the digital imagedoes not meet or exceed the predetermined minimum included segmentthreshold, increasing the minimum color distance threshold by apredetermined amount: clustering all pairs of neighboring segmentshaving a color value distance less than or equal to the increasedminimum color distance threshold; computing a new connected structurebased on a shape of the clustered segments; computing a new polygonbounding the connected structure; comparing a fraction of segmentsincluded in both the new connected structure and the new polygon,relative to a total number of segments in the digital image, to theminimum included segment threshold; and in response to determining thefraction of segments included in both the new connected structure andthe new polygon, relative to the total number of segments in the digitalimage meets or exceeds the minimum included segment threshold, croppingthe digital image based on edges of the polygon.
 3. The method asrecited in claim 2, comprising, in response to determining the fractionof segments included in both the new connected structure and the newpolygon, relative to the total number of segments in the digital imagedoes not meet or exceed the predetermined minimum included segmentthreshold, iteratively further increasing the increased minimum colordistance threshold, and repeating the clustering, the computing of newconnected structure(s), the computing of new polygon(s), and thecomparing of fractions of segments in the new connected structure(s) andthe new polygon(s) to the minimum included segment threshold untildetermining the fraction of segments included in both the new connectedstructure and the new polygon, relative to the total number of segmentsin the digital image meets or exceeds the minimum included segmentthreshold; and in response to determining the fraction of segmentsincluded in both the new connected structure and the new polygon,relative to the total number of segments in the digital image meets orexceeds the minimum included segment threshold, cropping the digitalimage based on edges of the polygon.
 4. The method as recited in claim1, wherein each segment is characterized by a rectangular shape, andwherein each segment is characterized by having a same height and a samewidth.
 5. The method as recited in claim 1, wherein the color distancevalue is computed based on one or more measures selected from: anintensity value of one or more color channels for each of thecorresponding pixels; a mean intensity value across two or more colorchannels for each of the corresponding pixels; a median intensity valueof one or more color channels for each of the corresponding pixels; anda range of intensity value across one or more color channels for each ofthe corresponding pixels.
 6. The method as recited in claim 1, whereinthe corresponding pixels comprise: single center pixels from eachrespective pair of neighboring segments; a set of pixels immediatelysurrounding a central point of each respective pair of neighboringsegments; or the single center pixels from each respective pair ofneighboring segments and a set of pixels immediately surrounding thesingle center pixels from each respective pair of neighboring segments.7. The method as recited in claim 6, wherein the corresponding pixelscomprise: the single center pixels from each respective pair ofneighboring segments and pixels that are located cardinally adjacent thesingle center pixels; the single center pixels from each respective pairof neighboring segments and pixels that are located diagonally adjacentthe single center pixels; the single center pixels from each respectivepair of neighboring segments, pixels that are located cardinallyadjacent the single center pixels, and pixels that are locateddiagonally adjacent the single center pixels; or the single centerpixels from each respective pair of neighboring segments, pixels thatare located cardinally adjacent the single center pixels, pixels thatare located diagonally adjacent the single center pixels, and pixelsthat are located cardinally or diagonally adjacent to the pixels thatare located cardinally adjacent the single center pixels and/or thepixels that are located diagonally adjacent the single center pixels. 8.The method as recited in claim 6, comprising applying a weightedadjustment to a contribution of each of the corresponding pixels to thecolor distance value, wherein the weighted adjustment is based on: apixel distance between the corresponding pixel and a central point ofthe segment containing the corresponding pixel, and/or expected colorcharacteristics of the object; and wherein the color distance value iscomputed based on one or more measures selected from: an intensity valueof one or more color channels for each of the corresponding pixels; amean intensity value across two or more color channels for each of thecorresponding pixels; a median intensity value of one or more colorchannels for each of the corresponding pixels; and a range of intensityvalue across one or more color channels for each of the correspondingpixels.
 9. The method as recited in claim 1, wherein the neighboringsegments of the digital image comprise: segments that are locatedcardinally adjacent one another; segments that are located diagonallyadjacent one another; or segments that are located cardinally adjacentone another and segments that are located diagonally adjacent oneanother.
 10. The method as recited in claim 1, wherein the colordistance value is computed based on color information representedaccording to a CIELUV color space.
 11. The method as recited in claim 1,wherein computing the polygon comprises: estimating a plurality of edgepixels within the digital image; identifying a plurality of lines basedon the plurality of estimated edge pixels; forming a set of candidatepolygons from the plurality of identified lines; filtering the set ofcandidate polygons to remove one or more outlier candidate polygons,wherein the outlier candidate polygons are characterized by including atleast one vertex angle having a value outside a predetermined range ofexpected vertex angle values; computing a confidence score for eachcandidate polygon remaining in the set of candidate polygons afterremoving the outlier candidate polygons; and selecting the candidatepolygon having the highest confidence score as the polygon bounding theconnected structure.
 12. The method as recited in claim 11, wherein thepredetermine range of expected vertex angle values includes angle valuesfrom about 60 degrees to about 120 degrees.
 13. The method as recited inclaim 11, wherein the outlier candidate polygons are additionally oralternatively characterized by including at least one pair of oppositevertex angles having a sum value outside a predetermined range ofexpected summed opposite vertex angle values.
 14. The method as recitedin claim 13, wherein the predetermined range of expected summed oppositevertex angle values includes summed values from about 150 degrees toabout 210 degrees.
 15. The method as recited in claim 11, whereinestimating the plurality of edge pixels comprises a Canny edgeestimation.
 16. The method as recited in claim 11, wherein identifyingthe plurality of lines based on the plurality of estimated edge pixelscomprises a Hough transform.
 17. The method as recited in claim 11,wherein the confidence score for each candidate polygon is based on oneor more factors selected from: a binary score for one or more verticesof the candidate polygon; a polygon fitness to shape score; and/or aline confidence score for one or more lines of the candidate polygon.18. The method as recited in claim 17, wherein the binary score for agiven vertex has a value of zero if the given vertex angle ischaracterized by a value outside the predetermined range of expectedvertex angle values, or a value of one if the given vertex angle is avalue within the predetermined range of expected vertex angle values.19. The method as recited in claim 17, wherein the polygon fitness toshape score comprises a product of a plurality of summed squaredistances between each pair of opposite sides of the polygon.
 20. Themethod as recited in claim 1, wherein the minimum included segmentthreshold has a value of about 0.6.